Using Fortanix Data Security Manager for Secrets Injection in Kubernetes

1.0  Introduction

This article describes how to integrate Fortanix Data Security Manager (DSM) with Kubernetes and OpenShift for managing and securing application secrets. It also contains the information that a user requires to:

  • Encrypt Kubernetes secrets in etcd using Fortanix DSM Kubernetes KMS Plugin.
  • Install Fortanix DSM secrets injection controller in Kubernetes cluster for automatic secrets injection at runtime to applications.
  • Set up controller to authenticate using Fortanix DSM API Keys.
  • Set up controller to authenticate using Kubernetes Service Account JWT tokens.
  • Deploy an application with auto secrets injection enabled.

2.0  Background

Cloud-native and hybrid applications are increasingly deployed within containers. Kubernetes is transforming not only the application architecture but also the development and operations pipelines for application delivery.

Every day, teams come to rely on numerous secrets in their development and operational DevOps processes. Secrets ranging from passwords, tokens, certificates, SSH keys, and database credentials simply cannot be hardcoded or configured statically anymore.

Fortanix offers a powerful yet straightforward solution for enterprises and DevOps alike to centralize the storage and management of security artifacts. Secrets, cryptographic keys, and certificates can be extended to widely used applications like databases, PKI systems, HTTP servers/proxies, and custom applications. Most importantly it does so without sacrificing the security of an HSM and the agility of a massively scalable KMS built on a modern technology stack.

2.1  Secrets Management in Kubernetes

Kubernetes operates two realms commonly known as the control plane and data plane. Various components within the control plane are responsible for the orchestration of backbone services necessary to run a distributed application.

Kubernetes relies on internal secrets for bootstrapping various critical services in the control plane to ensure trust. These secrets are stored in etcd database and are simply base64 encoded. Fortanix DSM can encrypt the entirety of etcd data on behalf of Kubernetes to ensure data and application security at-rest. This is a critical requirement for operating any enterprise Kubernetes cluster. Encryption keys do not leave the HSM, as Kubernetes invokes Fortanix DSM  to encrypt or decrypt the secondary data keys using a master key. This is achieved using Kubernetes KMS Plugin as described in Section 3.0.

In addition to serving as an external KMS provider to Kubernetes, Fortanix DSM can also directly own and manage application-level secrets for Kubernetes. The secrets will never be stored at rest in local databases like etcd, instead, they injected directly into the pods at runtime. This document also illustrates on multiple ways of authentication with Fortanix DSM for access to secrets. See Section 4.0.

Fortanix DSM supports multiple authentication mechanisms. API Key is the standard mechanism to authenticate. This API Key will be saved as a Kubernetes secret in the cluster. Additionally, Kubernetes and Google Cloud Service Accounts can authenticate with a Kubernetes generated trustworthy signed JSON Web Token (JWT). This mechanism maintains the credentials independently from the application simply by relying on the PKI established within the orchestration control plane ensures better security.

As many enterprises find their DevOps migrating into the Cloud, and with Kubernetes increasingly provisioning various kinds of distributed deployments, the need for robust application secrets store as a supplement to Kubernetes etcd should be paramount.

3.0  Kubernetes KMS Plugin

Effective Kubernetes 1.10, a new feature called KMS Encryption Provider lets organizations bring their own KMS, for encrypting secrets stored in Kubernetes etcd database. Fortanix provides an open-source plugin that communicates with Kubernetes over a Unix Socket. At bootup or during orchestration, the API Server automatically generates the DEKs and encrypts them using a master key (KEK) stored in Fortanix DSM.

3.1  Prerequisites

  • Kubernetes cluster, Version 1.10 and above.
  • Support for OpenShift 4.1 and above.

3.2  Setup

The steps to encrypt Kubernetes secrets with Key(s) stored in Fortanix DSM are described in the Fortanix GitHub Repository   - Readme.

To deploy the Kubernetes cluster using Rancher refer to the Fortanix GitHub Repository   – Rancher Guide to apply the encryption configuration.

4.0  Kubernetes Secrets Injection

Instead of storing sensitive data as Kubernetes secrets, secrets can be directly injected to their workloads at runtime using Admission Controllers provided by Fortanix. This provides much tighter control of the sensitive data, and these secrets are not stored in any local database or filesystem at rest. Developers and operators only need to specify the application secrets that will be pulled in automatically through environment variables in the destination pods/containers.

4.1  Architecture

Admission Controllers were introduced in Kubernetes version 1.9 and made publicly available (beta) since version 1.13. They are commonly implemented as Validation or Mutation Webhooks that inspect, verify and/or alter the pod specification during entry into or update within the cluster. When implemented with a dedicated sidecar for reading and refreshing secrets from Fortanix DSM, secrets are securely injected into the application.

8.png
Figure 1: Automated Secrets Injection through the Admission controller

The Illustration above demonstrates the workflow of injection of secrets. Cluster Admin updates the API Server to accept Admission Controllers. A Mutating Webhook Configuration is applied, which runs a Kubernetes Service to inspect pod specifications and can be limited to either the default or a specific namespace. This mutation service is a simple web server that leverages the Kubernetes API to read the config maps, volumes and, containers of the pod seeking entry into or update within the cluster. Pod specifications need to have certain predefined annotations relevant to the mutation service. Optionally, the mutation service can be programmed to perform dynamic changes to the pod based on a flexible set of annotations.

This mutation service is a simple Web Server that is implemented in Go programming language supporting the Kubernetes API. This web server typically does the following –

  • The main method to keep the webserver daemon running.
  • HTTP endpoint at which requests from API Server are received, for example: /mutate
  • TLS certificates for mutual authentication and secure exchange of payload

The /mutate request typically invokes another package or function that performs a Kubernetes API Admission Review to:

  1. Validate if the request is compatible with the configuration
    • pod namespace
    • pod annotations
    • pod associated service account
  2. Mutate the request and dispatch the response such that pod specification includes:
    • Config maps for any variables needed by the sidecar containers
      • Environment variables
    • Volumes
      • Projected Service Account token if using JWT based authentication
    • Secrets, if needed
    • Sidecar Containers
      • Init Container
      • Refresh Container (optional)
    • Update pod annotations to reflect that:
      • Mutation or secrets injection is applied
      • Additional parameters passed to the sidecar or main app containers

Now, every time an application deployment or replica set attempts to create or update pods in the intended or target namespace, the Mutating Webhook will intercept and inject the Kubernetes artifacts (config maps, volumes, and, most importantly the sidecar containers). Since Fortanix DSM is a vault-less and agent-less KMS and Secrets repository, the init and refresh containers are simple commands instead of elaborate service daemons or processes. Either of these containers will authenticate with Fortanix DSM based on an API KEY or the trustworthy JWT generated by Kubernetes for the pod. Once authenticated, the sidecar containers pull secrets based on RBAC within Fortanix DSM identified by the API KEY or Kubernetes Service Account. Secrets are then injected directly into the pod and provisioned as environment variables. Ultimately, this allows applications to remain unaltered and their dependency on secrets seamlessly switches to Fortanix DSM from legacy repositories.

4.2  Prerequisites

  • Kubernetes cluster, Version 1.16 and above.
  • Also supports OpenShift 4.3 and above.
  • Fortanix DSM Account
    • Either on the Cloud or locally within your enterprise HSM.
  • Helm Version 3 and above.
    • Required for secrets injection controller installation.

5.0  Secrets Injection (API Based Authentication)

5.1  Fortanix DSM Configuration

  1. Set up a Group in your Fortanix DSM account for the purpose of storing the secrets.
  2. Create an App in the Group above. Note down the API Key for steps in the next section.
  3. Import Secrets in the above Group, which will be required by your Kubernetes applications.
    1. Import Security Object. Select type as “Secret”.
      NOTE
      Any type of Security Object can be injected and is not limited to Secrets.
    2. Use any name which is indicative of the secret. For Example: “org-xyz-db-oracle-admin-password”
    3. Import the secret value as RAW data type.
    4. Make sure to select “EXPORT” as one of the key operations.

5.2  Admission Controller In Kubernetes

5.2.1  Kubernetes (Cloud and Onpremise)

  1. Make sure the kube-config is setup to access your cluster.
  2. Make sure helm is installed.
  3. Download the Fortanix DSM admission controller helm chart from https://d2bzqwib4mjc49.cloudfront.net/dsm-secrets-injector-chart-2.0.tar.gz
  4. Extract the chart.
    $ tar -xf dsm-secrets-injector-chart-2.0.tar.gz
    • This will create a folder called dsm-secrets-injector-chart containing the helm chart details.
  5. Install the dependent charts.
    $ helm dep up dsm-secrets-injector-chart
  6. Install the admission controller chart:
    $ helm install dsm-secrets-injector ./dsm-secrets-injector-chart
    NAME: dsm-secrets-injector
    LAST DEPLOYED: Thu Jul 1 10:34:58 2021
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    kubernetes Integration with Fortanix SDKMS has been deployed successfully.
  7. Verify if admission controller pods are healthy:
    $ kubectl get pods --namespace Fortanix
    NAME                                         READY    STATUS       RESTARTS    AGE
    fortanix-cert-setup-5g92h                    0/1      Completed    0           4h21m
    fortanix-secrets-injector-59cc799797-gx9tq   1/1      Running      0           4h21m
    $ kubectl logs fortanix-secrets-injector-59cc799797-gx9tq --namespace fortanix
    2021/01/29 03:45:46 Configuration:
    2021/01/29 03:45:46 controllerConfigFile: /opt/fortanix/controller-config.yaml
    2021/01/29 03:45:46 webServerConfig.port: 8443
    2021/01/29 03:45:46 webServerConfig.certFile: /opt/fortanix/certs/cert.pem
    2021/01/29 03:45:46 webServerConfig.keyFile: /opt/fortanix/certs/key.pem
    2021/01/29 03:45:46 AuthTokenType: api-key
    2021/01/29 03:45:46 SecretAgentImage: registry.hub.docker.com/fortanix/k8s-sdkms-secret-agent:1.1

5.2.2  Openshift

  1. Make sure crc and oc is setup to access your cluster. The following command should successfully return the cluster information.
    oc cluster-info
  2. Make sure helm is installed.
  3. Download the Fortanix DSM admission controller helm chart from https://d2bzqwib4mjc49.cloudfront.net/dsm-secrets-injector-chart-2.0-openshift.tar.gz
  4. Extract the chart:
    $ tar -xf dsm-secrets-injector-chart-2.0.tar.gz
    This will create a folder called dsm-secrets-injector-chart containing the helm chart details.
  5. Install the dependent charts:
    $ helm dep up dsm-secrets-injector-chart
  6. You need to extract the OpenShift API Server CA certificate so that Fortanix DSM Mutating Webhook Controller can trust the incoming webhook requests from OpenShift.
    $ oc extract -n openshift-kube-controller-manager secret/csr-signer --to - --keys tls.crt 2>/dev/null | base64 -w 0
    The command above will output a long string which will be a base64 encoded CA certificate. Note the value. For example:
    LS0tLS1CRUdJTiB . . . VORCBDRVJUSUZJQ0FURS0tLS0tCg==
  7. Install the admission controller chart:
    $ helm install dsm-secrets-injector ./dsm-secrets-injector-chart --set global.caBundle=LS0tLS1CRUdJTiBVO … SUZJQ0FURS0tLS0tCg==
    NAME: dsm-secrets-injection
    LAST DEPLOYED: Thu Jul 1 10:34:58 2021
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    kubernetes Integration with Fortanix SDKMS has been deployed successfully.
    NOTE
    Alternatively, the file dsm-secrets-injector-chart/values.yaml can be edited to add the CA value to the key global.caBundle and then installed as:
    $ helm install dsm-secrets-injector ./dsm-secrets-injector-chart
  8. Verify if the admission controller pods are healthy:
    $ oc project Fortanix 
    Now using project "fortanix" on server "https://api.crc.testing:6443".
    $ oc status
       
    project fortanix on server https://api.crc.testing:6443
    svc/fortanix-secrets-injector-svc - 10.217.4.34:443 -> 8443 deployment/fortanix-secrets-injector deploys fortanix/k8s-sdkms-secrets-injector:1.0 deployment no.1 running for 4 minutes - 1 pod job/fortanix-cert-setup manages fortanix/k8s-sdkms-cert-setup:1.0 created 4 minutes ago 1/1 completed 0 running 2 infos identified, use 'oc status --suggest' to see details.
    $oc get pods
    fortanix-cert-setup-5g92h                    0/1     Completed   0          4h21m
    fortanix-secrets-injector-59cc799797-gx9tq   1/1     Running     0          4h21m
    
    $ oc logs fortanix-secrets-injector-59cc799797-gx9tq
    2021/01/29 03:45:46   Configuration:
    2021/01/29 03:45:46   controllerConfigFile: /opt/fortanix/controller-config.yaml
    2021/01/29 03:45:46   webServerConfig.port: 8443
    2021/01/29 03:45:46   webServerConfig.certFile: /opt/fortanix/certs/cert.pem
    2021/01/29 03:45:46   webServerConfig.keyFile: /opt/fortanix/certs/key.pem
    2021/01/29 03:45:46   AuthTokenType: api-key
    2021/01/29 03:45:46   SecretAgentImage: registry.hub.docker.com/fortanix/k8s-sdkms-secret-agent:ubi7-1.1
    

5.3  Enable Auto Secrets Injection in Kubernetes Namespace

To enable auto-injection of secrets in Kubernetes application, the following modifications are required to existing application setup:

  1. Update the Kubernetes namespace/OpenShift project used by the application, to add a new label fortanix-secrets-injector: enabled. This enables the namespace/project for automated secrets injection. For example, if your applications are running in a namespace called demo-apps (This can be the default namespace too), edit the namespace as follows:

    Kubernetes:
    $ kubectl label namespace demo-apps fortanix-secrets-injector=enabled
    Openshift:
    $ oc label namespace demo-apps fortanix-secrets-injector=enabled
  2. Add a secret in the demo-apps namespace which will store the Fortanix DSM authentication parameters. The following are the required parameters:

    api_endpoint
    API Endpoint is the same as your Fortanix DSM deployment endpoint. For example: https://sdkms.fortanix.com
    api_key
    API Key is the same as copied from Fortanix DSM Dashboard from step 5.0.2.
    name
    The name of the secret is prefixed by the serviceAccount of the target application. In the following example, demo-sa is the serviceAccount used by the application, hence the name is demo-sa-sdkms-credentials. If serviceAccount is not set explicitly by the application, then use the value default (the name of secret would then be default-sdkms-credentials)
    cacert
    Fortanix DSM CA Certificate. This is required only if the Fortanix DSM endpoint is served using a custom organization CA. (Optional).
    The secret needs to be created separately for each namespace and serviceAccount used by the applications.

    Kubernetes:
    $ kubectl create secret generic demo-sa-sdkms-credentials --from-file=cacert=dsm-ca.crt --from-literal api_endpoint=https://sdkms.fortanix.com --from-literal api_key=NCNNJIEDN....03ninCWW0i0n -n demo-apps
    OpenShift:
    $ oc create secret generic demo-sa-sdkms-credentials --from-file=cacert=dsm-ca.crt --from-literal api_endpoint=https://sdkms.fortanix.com --from-literal api_key=NCNNJIEDN....03ninCWW0i0n -n demo-apps
  3. Update your application pod/deployment/statefulset/daemonset specification to add annotations for enabling secrets injection.
    1. One can inject secrets either as
      1. Environment Variables
      2. Files mounted at a particular location.
    2. Multiple secrets can be mounted at the same time.
    3. You will need to restart existing pods for annotations to take effect. Once added, any restart of the deployment/pods will automatically inject the secrets.
    4. See Section 7.0 on how to update application annotations for auto secrets injection.

6.0  Secrets Injection (Auto-Auth Using Trustworthy Tokens)

Kubernetes 1.12 released a new feature called Service Account Token Volume Projection.

This allows for applications to rely on named service accounts and an additional layer of security for authentication and authorization. This also solves the problem of Secret Zero by using Kubernetes provided JWT tokens for authentication (instead of API Key)

Upon deployment, the API Server dynamically injects the service account identity into the pod(s) during runtime to provide an attestation. The identity provides a verifiable way for external applications or third parties to inspect a signed token and extend to it additional permissions to resources. Applications running within separate containers inside the same pod will have access to the signed token. The signing of identity tokens is configurable by the Cluster Admin by patching the API Server. This diagram illustrates the flow of the signed identity token from the initial assignment to re-issuance or revocation.

AuthTokenFlow.png Figure 2: Authentication Token Flow

When Fortanix DSM receives and verifies the token using the public key of the Kubernetes API Server, it can then validate additional roles and access controls for its native resources. Since these signed tokens have a finite expiration period and are rotated automatically by Kubernetes based on the Pod spec, capturing of a single token has limited use, thereby drastically reducing the threat surface. Figure 3 also shows the authentication scheme established within Kubernetes for the API Server, the Kubelet, the deployment pods, and the external application. Since the signed token is generated from the service account, which itself is encrypted at rest, there is no storage of the JWT during injection by Kubelet into the pod.

7.pngFigure 3: Authentication using trustworthy Flow

6.1  Security Principles

Each service account in Kubernetes gets a default token. These tokens are signed by the API Server, but not rotated and have unlimited validity during their existence. Using these tokens is perhaps equivalent to a legacy password. Kubernetes took note and created the token volume projection to address the security gaps. These signed tokens are rotated automatically based on pod specifications and their validity is limited in scope based on attributes outlined below:

  • Associated with a single Kubernetes service account.
  • Permissioned by the cluster role and role bindings.
  • Identified by an issuer name or URL.
  • Distributed to a specific audience for whom the token is generated and targeted.
  • Invalidated outside of its time window through a Not-Before or Valid-After TTL.
  • Signed by the API Server private key and verifiable by its public key.

Based on these tenets, the signed tokens (JWT) are more trustworthy than standard service account tokens and when automatically authorized by Fortanix DSM, renders applications access to their secrets, keys, or certificates in a seamless manner. This tremendously improves the authentication of applications, services, and/or machines by removing the need to manage individual identities.

6.2  Kube API Server Update for Signing Key

Depending upon the Kubernetes cluster, this step can require a primitive configuration change in the API Server (/etc/kubernetes/manifest/kube-apiserver.yaml).

Audience
This value should be the Fortanix DSM endpoint URL.
Issuer
Denotes your cluster/application. This will be used to identify the tokens for this cluster when pods authenticate with Fortanix DSM for secrets injection.
Signing Key File
Is the signing key selected for token generation. The public part of this key will be used by Fortanix DSM to verify the tokens. One can create a new signing key using openssl as:
openssl genrsa -out token-signing.key 2048

6.2.1  Minikube

Start Minikube with these additional flags to update the API Server:

minikube start \
--extra-config=apiserver.service-account-issuer=https://k8s.test \
--extra-config=apiserver.service-account-api-audiences=https://sdkms.fortanix.com \
--extra-config=apiserver.service-account-signing-key-file=/path/to/k8s/pki/token-signing.key

6.2.2 Docker Setup

Docker Desktop requires editing kube-apiserver.yaml manifest by accessing the master or host node:

--service-account-issuer=https://token-issuing.application.com
--service-account-api-audiences=https://sdkms.fortanix.com
--service-account-signing-key-file=/path/to/k8s/pki/token-signing.key

Here is a Docker Desktop command to get remote access to the master node:

docker run --rm -it --privileged --pid=host walkerlee/nsenter \
-t 1 -m -- vi /etc/kubernetes/manifests/kube-apiserver.yaml

Once the API server is configured, verify that the kube-system pods are running.

NOTE
  • If the Kubernetes controller-manager is in CrashbackLoopOff, then it is likely failing to authenticate during bootstrap. Verify that the service-account-signing-key-file matches the service-account-key-file specified in the kube-controller-manager.yaml.
  • If the service-account-issuer has extra trailing characters such as a forward slash (/) it will fail authentication with Fortanix DSM.

6.3  Signing Key Extraction For Fortanix DSM

The public key corresponding to the signing key of the Kubernetes API Server is needed so Fortanix DSM can verify the JWT upon receipt of an authentication request.

-- service-account-signing-key-file=/path/to/k8s/pki/token-signing.key

Also, if the pub file is not present within the Kubernetes PKI folder, then run the following command to get the public key.

openssl rsa -in /path/to/k8s/pki/token-signing.key -pubout > key.pub

6.4  Fortanix DSM Configuration for Auto-Auth

  1. Set up a group in your Fortanix DSM account for the purpose of storing the secrets.
  2. Create an app in the above Group with the following parameters:

    Name
    Should be exactly as system:serviceaccount:default:<serviceaccountname> For example: system:serviceaccount:default:default (for default service account).
    Authentication Method
    JSON Web Token.
    Valid Issuer(s)
    Should match the specification of Kubernetes API Server (Section 6.2).
    -- service-account-issuer=https://token-issuing.application.com
  3. Upload the Public Key (and Key ID if exists) obtained from Section 6.3. DSMApp.png
    Figure 4: Fortanix DSM application created The illustration above shows a Fortanix DSM App corresponding to the Kubernetes Service Account.
    Details of the Fortanix DSM App attributes are shown on the next page. The public key extracted in the previous step can also be dynamically fetched for verification. This supports auto-TLS if it is set up in Kubernetes, wherein the keys and certificates are refreshed more frequently. AddApp.png
    Figure 5: Add new app
  4. Copy the Account ID. Go to Account Settings and copy the account ID provided there.
  5. For Secrets injection use-case, import Secrets in the above Group, which will be required by your Kubernetes applications.
    1. Import Security Object. Select type as “Secret”.
      Note: Any type of Security Object can be injected and is not limited to Secrets.
    2. Use any name which is indicative of the secret. For example: “org-xyz-db-oracle-admin-password”.
    3. Import the secret value as RAW data type.
    4. Make sure to select “EXPORT” as one of the key operations.

6.5  Admission Controller in Kubernetes

  1. Make sure the kube-config is set up to access your cluster.
  2. Make sure helm is installed.
  3. In the same environment where kube-config is set, download the Fortanix DSM admission controller helm chart from https://d2bzqwib4mjc49.cloudfront.net/dsm-secrets-injection-chart-1.2.tar.gz
  4. Extract the chart:
    $ tar -xf dsm-secrets-injection-chart-1.2.tar.gz
    This will create a folder called secrets-injection-chart containing the helm chart details.
  5. Install the dependent charts:
    $ helm dep up sdkms-secrets-injection
  6. Install the admission controller chart. Use the following parameters:
    • configmap.authTokenType = jwt
    • configmap.tokenVolumeProjection.audience = https://sdkms.fortanix.com
      • This needs to be the same as the Fortanix DSM endpoint
    • [Optional] tokenVolumeProjection.expirationSeconds = 600
      • In seconds. Default value is 3600, that is, 1 hour)
    $ helm install sdkms-secrets-injection ./sdkms-secrets-injection –-set configmap.authTokenType=jwt,configmap.tokenVolumeProjection.audience=https://sdkms.fortanix.com
  7. Verify if admission controller pods are healthy.
    $ kubectl get pods --namespace Fortanix
    NAME                                         READY   STATUS      RESTARTS   AGE
    fortanix-cert-setup-5g92h                    0/1     Completed   0          4h21m
    fortanix-secrets-injector-59cc799797-gx9tq   1/1     Running     0          4h21m  
    $ kubectl logs fortanix-secrets-injector-59cc799797-gx9tq --namespace fortanix
    2021/01/29 03:45:46   Configuration:
    2021/01/29 03:45:46   controllerConfigFile: /opt/fortanix/controller-config.yaml
    2021/01/29 03:45:46   webServerConfig.port: 8443
    2021/01/29 03:45:46   webServerConfig.certFile: /opt/fortanix/certs/cert.pem
    2021/01/29 03:45:46   webServerConfig.keyFile: /opt/fortanix/certs/key.pem
    2021/01/29 03:45:46   AuthTokenType: jwt
    2021/01/29 03:45:46   SecretAgentImage: fortanix/k8s-sdkms-secret-agent:1.0
    2021/01/29 03:45:46   TokenVolumeProjection:
    2021/01/29 03:45:46   AddToAllPods = true
    2021/01/29 03:45:46   Audience = https://sdkms.fortanix.com
    2021/01/29 03:45:46   ExpirationSeconds = 3600

6.6  Enable Auto Secrets Injection in Kubernetes Namespace

To enable auto-injection of secrets in Kubernetes application, the following modifications are required to existing application setup:

  1. Update the Kubernetes namespace/OpenShift project used by the application, to add a new label fortanix-secrets-injector: enabled. This enables the namespace/project for automated secrets injection.
    • For example: If your applications are running in a namespace called demo-apps (This can be the default namespace too), edit the namespace as follows: Kubernetes:
      $ helm install sdkms-secrets-injection ./sdkms-secrets-injection –-set configmap.authTokenType=jwt,configmap.tokenVolumeProjection.audience=https://sdkms.fortanix.com
      Kubernetes:
      $ kubectl edit namespace demo-apps
      apiVersion: v1
      kind: Namespace
      metadata:
      creationTimestamp: "2021-01-29T03:54:57Z"
      labels:
      fortanix-secrets-injector: enabled
      name: fortanix-demo
      resourceVersion: "139438"
      selfLink: /api/v1/namespaces/demo-apps
      uid: 1b4b0a68-27ab-44de-aea1-48300d0d2255
      spec:
      finalizers:
      - kubernetes
      status:
      phase: Active
      Openshift:
      $ oc label namespace demo-apps fortanix-secrets-injector=enabled
      namespace/demo-apps labeled
  2. Add a secret in the demo-apps namespace which will store the Fortanix DSM authentication parameters.
    • The name of this secret needs to match the ServiceAccount used by target application pod/deployment.
    • Create a file called sdkms-cred.yaml with following content:
      apiVersion: v1
      kind: Secret
      metadata:
      name: demo-app-sa-sdkms-credentials
      namespace: demo-apps
      type: Opaque
      stringData:
      api_endpoint: https://sdkms.fortanix.com
      account_id: NzhkO…Ghn
    • API Endpoint is the same as your Fortanix DSM deployment endpoint.
    • Account ID is the same as Fortanix DSM Account ID. This can be found in the Settings section of your Fortanix DSM account.
    • The name is prefixed by  serviceAccount of the target application. In the example, demo-app-sa is the serviceAccount used by the application. If serviceAccount is not set explicitly by the application, then use the value default (the name of secret would then be default-sdkms-credentials)
    • Create the secret.
      Kubernetes:
      $ kubectl create -f sdkms-cred.yaml
      OpenShift:
      $ oc create -f sdkms-cred.yaml
    • This secret needs to be created separately for each namespace and serviceAccount used by the applications.
  3. Update your application pod/deployment/statefulset/daemonset specification to add annotations for enabling secrets injection.
    1. One can inject secrets either as
      1. Environment Variables
      2. Files mounted at a particular location
    2. Multiple secrets can be mounted at the same time.
    3. You will need to restart existing pods for annotations to take effect. Once added, any restart of the deployment/pods will automatically inject the secrets.
    4. See Section 7.0 on how to update application annotations for auto secrets injection.

7.0 Annotations for Auto Secrets in Injection

7.1  Annotations for File-Based Secrets Injections

This section describes the annotations required in application pod/deployment for secrets to be injected as files.

  • An emptyDir type Kubernetes volume is mounted by the init container to the application container(s) in a pod.
  • This volume hosts all the secrets injected as separate files. This mount path is configurable.
  • This mounted volume is owned by root:root and file permissions are set to 750. Non-root user or lesser permissions is not yet supported.

    The following example will mount two secrets on the pods, at the file locations
  • /opt/myapp/credentials/xyzpassword.txt
  • /opt/myapp/credentials/abcpassword.txt
    apiVersion: v1
    kind: Pod
    metadata:
    namespace: demo-apps
    name: sample-pod
    annotations:
    secrets-injector.fortanix.com/inject-through-environment: "false"
    secrets-injector.fortanix.com/secrets-volume-path: /opt/myapp/credentials
    secrets-injector.fortanix.com/inject-secret-xyzpassword.txt: "org-xyz-db-oracle-admin-password"
    secrets-injector.fortanix.com/inject-secret-abcpassword.txt: "org-abc-db-oracle-admin-password"
    spec:
    containers:
    - name: myapp
    image: myapp:1.0
  • secrets-injector.fortanix.com/secrets-volume-path is folder where all secrets will save as files. If not set, the default location is /var/run/fortanix/secrets in the target pod.
  • The annotation for specifying a Secret to be injected is: secrets-injector.fortanix.com/inject-secret-<secret-mount-file>: <SDKMS Security Object name>
    • <secret-mount-file> is the file name to be mounted in the above folder which will contain the secret value upon injection.
    • <SDKMS Security Object name> is the name of Fortanix DSM Security Object which was imported in step 4.3.1.
    • Multiple such entries can be added for injecting multiple secrets.

7.2  Annotations for Environment Variables Based Secrets Injections

This section describes the annotations required in application pod/deployment for secrets to be injected as environment variables.

Two emptyDir type Kubernetes volumes are mounted by the init container to the application container(s) in a pod.

  • Volume mounted at /var/run/fortanix/secrets hosts all secrets as separate files.
  • Volume mounted at /opt/fortanix contains a script env-wrapper which is used to load these secrets as environment variables in the main process.
NOTE
In order to load the secrets as environment variables, the admission controller patches the pod’s entry-point command to run env-wrapper first and then continue with the original entry-point command.

Hence this mode of secret injection is supported only if the pod/deployment spec explicitly defines the container command and do not rely of docker image’s default entry-point.

  • The following example will mount two secrets as environment variables XYZ_DB_PASS and ABC_DB_PASS.
    apiVersion: v1
    kind: Pod
    metadata:
    namespace: demo-apps
    name: sample-pod
    annotations:
    secrets-injector.fortanix.com/inject-through-environment: "true"
    secrets-injector.fortanix.com/inject-secret-XYZ_DB_PASS: "org-xyz-db-oracle-admin-password"
    secrets-injector.fortanix.com/inject-template-XYZ_DB_PASS: "export {{.Key}}=\"{{.Value}}\""
    secrets-injector.fortanix.com/inject-secret-ABC_DB_PASS: "org-abc-db-oracle-admin-password"
    secrets-injector.fortanix.com/inject-template-ABC_DB_PASS: "export {{.Key}}=\"{{.Value}}\""

    spec:
    containers:
    - name: myapp
    image: myapp:1.0
    command:
    - /bin/myapp
    args:
    - "--flag=true"
  • The annotation for specifying a Secret to be injected is:(there are two annotations per secret to be injected):
    • secrets-injector.fortanix.com/inject-secret-<secret-mount-file>: <SDKMS Security Object name>
    • secrets-injector.fortanix.com/inject-template<secret-env-variable>: "export {{.Key}}=\"{{.Value}}\""
      • <secret-env-variable> is the environment variable that will contain the secret value upon injection.
      • <SDKMS Security Object name> is the name of the Fortanix DSM Security Object that was imported in Section 6.4.
      • The second annotation above is a template based on the container environment to export the value as an env variable. Use the provided template for Linux containers. For non-Linux containers, the template may require a modification.

8.0  Conclusion

DevOps is fast transitioning into “DevSecOps”. Security teams are tasked during application development to model network topology as well as protect the credentials and secrets used to provision infrastructure, deploy applications and/or databases. Passwords and API tokens are constantly spread out within the enterprise and often stored on a disk using configuration files or data bags. Security typically falls to dedicated roles but involves all other teams that are delivering an application.

As a reference, the diagram on the following page illustrates the use of the Kubernetes Service Account Token Projection in AWS. It is used to tailor a fine-grained RBAC (role-based-access-control) mechanism in conjunction with the AWS IAM (identify-and-authorization-management) for its EKS offering.

8.1  Other Examples

More details are available at kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection

AWS guide on the provisioning of fine-grained IAM roles for Kubernetes service accounts: aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

Was this article helpful?
0 out of 0 found this helpful