Using Fortanix Data Security Manager for Secrets Injection in Kubernetes

1.0  Introduction

This article describes how to integrate Fortanix Data Security Manager (DSM) with Kubernetes and OpenShift for managing and securing application secrets. It also contains the information that a user requires to: 

• Encrypt Kubernetes secrets in etcd using Fortanix DSM Kubernetes KMS Plugin. 
• Install Fortanix DSM secrets injection controller in Kubernetes cluster for automatic secrets injection at runtime to applications. 
• Set up controller to authenticate using Fortanix DSM Application Processing Interface (API) Keys. 
• Set up controller to authenticate using Kubernetes Service Account JSON Web Tokens (JWTs). 
• Deploy an application with auto secrets injection enabled. 

2.0  Background

Cloud-native and hybrid applications are increasingly deployed within containers. Kubernetes is transforming not only the application architecture but also the development and operations pipelines for application delivery.

Every day, teams come to rely on numerous secrets in their development and operational DevOps processes. Secrets ranging from passwords, tokens, certificates, Secure Shell (SSH) keys, and database credentials simply cannot be hardcoded or configured statically anymore.

Fortanix offers a powerful yet straightforward solution for enterprises and DevOps alike to centralize the storage and management of security artifacts. Secrets, cryptographic keys, and certificates can be extended to widely used applications like databases, Public Key Infrastructure (PKI) systems, HyperText Transfer Protocol (HTTP) servers/proxies, and custom applications. Most importantly, it does so without sacrificing the security of a Hardware Security Module (HSM) and the agility of a massively scalable Google Cloud Key Management Service (KMS) built on a modern technology stack.

2.1  Secrets Management in Kubernetes

Kubernetes operates two realms commonly known as the control plane and data plane. Various components within the control plane are responsible for the orchestration of backbone services necessary to run a distributed application.

Kubernetes relies on internal secrets for bootstrapping various critical services in the control plane to ensure trust. These secrets are stored in etcd database and are simply base64 encoded. Fortanix DSM can encrypt the entirety of etcd data on behalf of Kubernetes to ensure data and application security at-rest. This is a critical requirement for operating any enterprise Kubernetes cluster. Encryption keys do not leave the HSM, as Kubernetes invokes Fortanix DSM to encrypt or decrypt the secondary data keys using a master key. This is achieved using Kubernetes KMS Plugin as described in Section 3.0.

In addition to serving as an external KMS provider to Kubernetes, Fortanix DSM can also directly own and manage application-level secrets for Kubernetes. The secrets are never stored at rest in local databases like etcd, instead, they are injected directly into the pods at runtime. This document also illustrates on multiple ways of authentication with Fortanix DSM for access to secrets. See Section 4.0.

Fortanix DSM supports multiple authentication mechanisms. API Key is the standard mechanism to authenticate. This API Key is saved as a Kubernetes secret in the cluster. Additionally, Kubernetes and Google Cloud Service Accounts can authenticate with a Kubernetes generated trustworthy signed JWT. This mechanism maintains the credentials independently from the application simply by relying on the PKI established within the orchestration control plane ensures better security.

As many enterprises find their DevOps migrating into Cloud, and with Kubernetes increasingly provisioning various kinds of distributed deployments, the need for robust application secrets stored as a supplement to Kubernetes etcd should be paramount.

3.0  Kubernetes KMS Plugin

Effective Kubernetes 1.10, a new feature called KMS Encryption Provider lets organizations bring their own KMS, for encrypting secrets stored in Kubernetes etcd database. Fortanix provides an open-source plugin that communicates with Kubernetes over a Unix Socket. At bootup or during orchestration, the API Server automatically generates the DEKs and encrypts them using a master key (KEK) stored in Fortanix DSM.

3.1  Prerequisites

• Kubernetes cluster, Version 1.10 and above.
• Support for OpenShift 4.1 and above.
• Access to Kubernetes Master nodes is required.

3.2  Setup

The steps to encrypt Kubernetes secrets with Key(s) stored in Fortanix DSM is described in the Fortanix GitHub Repository - Readme.

To deploy the Kubernetes cluster using Rancher, refer to the Fortanix GitHub Repository – Rancher Guide to apply the encryption configuration.

4.0  Kubernetes Secrets Injection - Architecture

Instead of storing sensitive data as Kubernetes secrets, secrets can be directly injected to their workloads at runtime using Admission Controllers provided by Fortanix. This provides much tighter control of the sensitive data, and these secrets are not stored in any local database or file system at rest. Developers and operators only need to specify the application secrets that will be pulled in automatically through environment variables in the destination pods/containers.

Admission Controllers were introduced in Kubernetes version 1.9 and made publicly available (beta) since version 1.13. They are commonly implemented as Validation or Mutation Webhooks that inspect, verify and/or alter the pod specification during entry into or update within the cluster. When implemented with a dedicated sidecar for reading and refreshing secrets from Fortanix DSM, secrets are securely injected into the application. Cluster Admin updates the API Server to accept Admission Controllers. A Mutating Webhook Configuration is applied, which runs a Kubernetes Service to inspect pod specifications and can be limited to either the default or a specific namespace. This mutation service is a simple web server that leverages the Kubernetes API to read the config maps, volumes and, containers of the pod seeking entry into or update within the cluster. Pod specifications need to have certain predefined annotations relevant to the mutation service. Optionally, the mutation service can be programmed to perform dynamic changes to the pod based on a flexible set of annotations.

This mutation service is a simple Web Server that is implemented in Go programming language supporting the Kubernetes API. This web server typically does the following:

1. The main method to keep the webserver daemon running.
2. HTTP endpoint at which requests from API Server are received, for example: /mutate
3. Transport Layer Security (TLS) certificates for mutual authentication and secure exchange of payload.

The /mutate request typically invokes another package or function that performs a Kubernetes API Admission Review to:

1. Validate if the request is compatible with the configuration

• pod namespace
• pod annotations
• pod associated service account

2. Mutate the request and dispatch the response such that pod specification includes:

• Config maps for any variables needed by the sidecar containers
  • Environment variables
• Volumes
  • Projected Service Account token if using JWT based authentication
• Secrets, if needed
• Sidecar Containers
  • Init Container
  • Refresh Container (optional)
• Update pod annotations to reflect that:
  • Mutation or secrets injection is applied
  • Additional parameters have been passed to the sidecar or main app containers

Now, every time an application deployment or replica sets attempts to create or update pods in the intended or target namespace, the Mutating Webhook intercepts and injects the Kubernetes artifacts (config maps, volumes, and, most importantly the sidecar containers). Since Fortanix DSM is a vault-less and agent-less KMS and Secrets repository, the init and refresh containers are simple commands instead of elaborate service daemons or processes. Either of these containers will authenticate with Fortanix DSM based on an API KEY or the trustworthy JWT generated by Kubernetes for the pod. Once authenticated, the sidecar containers pull secrets based on RBAC within Fortanix DSM identified by the API KEY or Kubernetes Service Account. Secrets are then injected directly into the pod and provisioned as environment variables. Ultimately, this allows applications to remain unaltered and their dependency on secrets seamlessly switches to Fortanix DSM from legacy repositories.

5.0  Secrets Injection Admission Controller

5.1  Prerequisites

• Kubernetes cluster, Version 1.16 and above.
• Also supports OpenShift 4.3 and above.
• Fortanix DSM Account
  • Either on the Cloud or locally within your enterprise HSM.
• Helm Version 3 and above.
  • Required for secrets injection controller installation.

5.2  Setup

5.2.1  Kubernetes (Cloud and On-premises)

1. Make sure the kube-config is set up to access your cluster.
2. Download the Fortanix DSM admission controller helm chart from
   https://download.fortanix.com/clients/dsm-secrets-injection/dsm-secrets-injector-chart-2.7.tar.gz 
3. Extract the chart.

   $ tar -xf dsm-secrets-injector-chart-2.7.tar.gz

   1. This creates a folder called dsm-secrets-injector-chart containing the helm chart details.
4. Install the admission controller chart:

   $ helm install dsm-secrets-injector ./dsm-secrets-injector-chart

   NAME: dsm-secrets-injector
   LAST DEPLOYED: Thu Jul 1 10:34:58 2021
   NAMESPACE: default
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   NOTES:
   kubernetes Integration with Fortanix DSM has been deployed successfully.

5. Verify if admission controller pods are healthy:

   $ kubectl get pods --namespace fortanix
   NAME                                         READY   STATUS      RESTARTS   AGE
   fortanix-cert-setup-5g92h                    0/1     Completed   0          4h21m
   fortanix-secrets-injector-59cc799797-gx9tq   1/1     Running     0          4h21m
   $ kubectl logs fortanix-secrets-injector-59cc799797-gx9tq --namespace fortanix
   2021/01/29 03:45:46 Configuration:
   2021/01/29 03:45:46   controllerConfigFile: /opt/fortanix/controller-config.yaml
   2021/01/29 03:45:46   webServerConfig.port: 8443
   2021/01/29 03:45:46   webServerConfig.certFile: /opt/fortanix/certs/cert.pem
   2021/01/29 03:45:46   webServerConfig.keyFile: /opt/fortanix/certs/key.pem
   2021/01/29 03:45:46 AuthTokenType: api-key
   2021/01/29 03:45:46 SecretAgentImage: registry.hub.docker.com/fortanix/k8s-sdkms-secret-agent:1.1
   

5.2.2 Openshift

1. Make sure crc and oc is set up to access your cluster. The following command should successfully return the cluster information:

   oc cluster-info 

2. Make sure helm is installed.
3. Download the Fortanix DSM admission controller helm chart from
   https://download.fortanix.com/clients/dsm-secrets-injection/dsm-secrets-injector-chart-2.7-openshift.tar.gz 
4. Extract the chart:

   $ tar -xf dsm-secrets-injector-chart-2.7-openshift.tar.gz

   This creates a folder called dsm-secrets-injector-chart containing the helm chart details.

5. Install the admission controller chart:

   $ helm install dsm-secrets-injector ./dsm-secrets-injector-chart

   NAME: dsm-secrets-injection
   LAST DEPLOYED: Thu Jul 1 10:34:58 2021
   NAMESPACE: default
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   NOTES: kubernetes Integration with Fortanix DSM has been deployed successfully.

6. Verify if the admission controller pods are healthy:

   $ oc project fortanix
   
   Now using project "fortanix" on server "https://api.crc.testing:6443".

   $ oc status
   In project fortanix on server https://api.crc.testing:6443
   svc/fortanix-secrets-injector-svc - 10.217.4.34:443 -> 8443 
   deployment/fortanix-secrets-injector deploys fortanix/k8s-sdkms-secrets-injector:1.0
   deployment 1 running for 4 minutes - 1 pod
   job/fortanix-cert-setup manages fortanix/k8s-sdkms-cert-setup:1.0
   created 4 minutes ago 1/1 completed 0 running
   2 infos identified, use 'oc status --suggest' to see details.
   

   $ oc get pods
   secrets-injector-cert-setup-5g92h                    0/1     Completed   0          4h21m
   fortanix-secrets-injector-59cc799797-gx9tq   1/1     Running     0          4h21m      

   $ oc logs fortanix-secrets-injector-59cc799797-gx9tq
   2021/01/29 03:45:46 Configuration:
   2021/01/29 03:45:46 controllerConfigFile: /opt/fortanix/controller-config.yaml
   2021/01/29 03:45:46 webServerConfig.port: 8443
   2021/01/29 03:45:46 webServerConfig.certFile: /opt/fortanix/certs/cert.pem
   2021/01/29 03:45:46 webServerConfig.keyFile: /opt/fortanix/certs/key.pem
   2021/01/29 03:45:46 AuthTokenType: api-key
   2021/01/29 03:45:46 SecretAgentImage: registry.hub.docker.com/fortanix/k8s-sdkms-secret-agent:ubi7-2.3
   

5.3  Troubleshooting and Maintenance

Admission controller runs as a standalone webservice, with the sole purpose of mutating pod specs served Kubernetes API server. Please note the following aspects of the webservice:

1. It never reaches out to Fortanix DSM or any other external entity.
2. It never handles any of the secrets directly. Controller only patches pods with information about Fortanix DSM init container and metadata of secrets to inject. Actual secrets are managed by init containers only.
3. Failure of admission controller webservice affects only new pod creation or restarts. Existing pods, which have secrets already injected, are not affected.

5.3.1  Multiple Replicas

It is advisable to run multiple instances of controller in production, for better resiliency. Do the following while installing the helm chart:

$ helm install dsm-secrets-injector ./dsm-secrets-injector-chart --set injector.replicas=3

5.3.2  Web Service TLS Expiration

The admission controller is a webservice over TLS. By default, it uses a self-signed certificate with a validity of 1 year. One can change the validity if needed as following during installation.

$ helm install dsm-secrets-injector ./dsm-secrets-injector-chart --set injector.certExpiration=730 (if to keep for 2 years)

5.3.3  Upgrade

If you wish to upgrade the controller to a newer released version, download the new helm chart and run the following helm upgrade command:

$ helm upgrade dsm-secrets-injector ./dsm-secrets-injector-chart-new-version

6.0  Secrets Injection (API based authentication)

6.1  Fortanix DSM Configuration

1. Set up a Group in your Fortanix DSM account for the purpose of storing the secrets.
2. Create an App in the Group above. Note down the API Key for steps in the next section.
3. Import Secrets in the above Group, which will be required by your Kubernetes applications.
   1. Import Security Object. Select type as “Secret”.
      NOTE

      Any type of Security Object can be injected and is not limited to Secrets.
   2. Use any name which is indicative of the secret. For Example: “org-xyz-db-oracle-admin-password”.
   3. Import the secret value as RAW data type.
   4. Make sure to select “EXPORT” as one of the key operations.

   6.2  Enable Auto Secrets injection in Kubernetes Namespace

   To enable auto-injection of secrets in Kubernetes application, following modifications are required in the existing application setup:

   1. Update the Kubernetes namespace/OpenShift project used by the application, to add a new label fortanix-secrets-injector: enabled. This enables the namespace/project for automated secrets injection.

      For example, if your applications are running in a namespace called demo-apps (This can be the default namespace too), edit the namespace as follows:

      Kubernetes:

      $ kubectl label namespace demo-apps fortanix-secrets-injector=enabled

       OpenShift:

      $ oc label namespace demo-apps fortanix-secrets-injector=enabled

   2. Add a secret in the demo-apps namespace which stores the Fortanix DSM authentication parameters. The following are the required parameters:
      1. api_endpoint: API Endpoint is the same as your DSM deployment endpoint. For example: https://sdkms.fortanix.com
      2. api_key: API Key is the same as copied from Fortanix DSM Dashboard.
      3. name: The name of the secret is prefixed by the serviceAccount of the target application. In the following example, demo-sa is the serviceAccount used by the application, hence the name is demo-sa-sdkms-credentials. If the serviceAccount is not set explicitly by the application, then use the value default (the name of secret would then be default-sdkms-credentials)
      4. cacert: Fortanix DSM CA Certificate. This is required only if the DSM endpoint is served using a custom organization CA. (Optional).
      5. The secret needs to be created separately for each namespace and serviceAccount used by the applications.

      Kubernetes:

      $ kubectl create secret generic
      demo-sa-sdkms-credentials --from-file=cacert=dsm-ca.crt
      --from-literal api_endpoint=https://sdkms.fortanix.com
      --from-literal api_key=NCNNJIEDN....03ninCWW0i0n
      -n demo-apps            

      OpenShift:

      $ oc create secret generic
      demo-sa-sdkms-credentials --from-file=cacert=dsm-ca.crt
      --from-literal api_endpoint=https://sdkms.fortanix.com
      --from-literal api_key=NCNNJIEDN....03ninCWW0i0n
      -n demo-apps            

   3. Update your application pod/deployment/statefulset/daemonset specification to add annotations for enabling secrets injection.
      1. One can inject secrets as:
         1. Environment Variables
         2. Files mounted at a particular location.
         3. Both env variables and mounted files.
         4. Multiple secrets can be mounted at the same time.
         5. You will need to restart existing pods for annotations to take effect. Once added, any restart of the deployment/pods will automatically inject the secrets.
         6. See Section 8.0 on how to update application annotations for auto secrets injection.

      7.0 Secrets Injection (Auto-Auth using Trustworthy Tokens)

      Kubernetes 1.12 released a new feature called Service Account Token Volume Projection.

      This allows for applications to rely on named service accounts and an additional layer of security for authentication and authorization. This also solves the problem of Secret Zero by using Kubernetes provided JWT tokens for authentication (instead of API Key).

       Upon deployment, the API Server dynamically injects the service account identity into the pod(s) during runtime to provide an attestation. The identity provides a verifiable way for external applications or third parties to inspect a signed token and extend to it additional permissions to resources. Applications running within separate containers inside the same pod will have access to the signed token. The signing of identity tokens is configurable by the Cluster Admin by patching the API Server.

      When Fortanix DSM receives and verifies the token using the public key of the Kubernetes API Server, it can then validate additional roles and access controls for its native resources. Since these signed tokens have a finite expiration period and are rotated automatically by Kubernetes based on the Pod spec, capturing of a single token has limited use, thereby drastically reducing the threat surface.

      Since the signed token is generated from the service account, which itself is encrypted at rest, there is no storage of the JWT during injection by Kubernetes into the pod.

      7.1  Security Principles

      Each service account in Kubernetes gets a default token. These tokens are signed by the API Server, but not rotated and have unlimited validity during their existence. Using these tokens is perhaps equivalent to using a legacy password. Kubernetes took note and created the token volume projection to address the security gaps. These signed tokens are rotated automatically based on pod specifications and their validity is limited in scope based on attributes outlined below:

      • Associated with a single Kubernetes service account.
      • Permissioned by the cluster role and role bindings.
      • Identified by an issuer name or URL.
      • Distributed to a specific audience for whom the token is generated and targeted.
      • Invalidated outside of its time window through a Not-Before or Valid-After Time to Live (TTL).
      • Signed by the API Server private key and verifiable by its public key.

      Based on these tenets, the signed tokens (JWT) are more trustworthy than standard service account tokens and when automatically authorized by Fortanix DSM, render applications access to their secrets, keys, or certificates in a seamless manner. This tremendously improves the authentication of applications, services and/or machines by removing the need to manage individual identities.

      7.2  Kube API Server Update for Signing Key

      Depending upon the Kubernetes cluster, this step can require a primitive configuration change in the API Server (/etc/kubernetes/manifest/kube-apiserver.yaml).

      • “Audience” value should be the Fortanix DSM endpoint URL.
      • “Issuer” to denote your cluster/application. This is used to identify the tokens for this cluster when pods authenticate with Fortanix DSM for secrets injection.
      • “Signing Key File” is the signing key selected for token generation. The public part of this key is used by Fortanix DSM to verify the tokens. One can create a new signing key using openssl as:

        openssl genrsa -out token-signing.key 2048

      7.2.1 Minikube

      Start Minikube with these additional flags to update the API Server:

      minikube start \
         --extra-config=apiserver.service-account-issuer=https://k8s.test \
         --extra-config=apiserver.service-account-api-audiences=https://sdkms.fortanix.com \
         --extra-config=apiserver.service-account-signing-key-file=/path/to/k8s/pki/token-signing.key
      

      7.2.2 Docker Desktop

      Docker Desktop requires editing kube-apiserver.yaml manifest by accessing the master or host node:

      --service-account-issuer=https://token-issuing.application.com
      --service-account-api-audiences=https://sdkms.fortanix.com
      --service-account-signing-key-file=/path/to/k8s/pki/token-signing.key

      Here is a Docker Desktop command to get remote access to the master node:

      docker run --rm -it --privileged --pid=host walkerlee/nsenter \
      -t 1 -m -- vi /etc/kubernetes/manifests/kube-apiserver.yaml        

      Once the API server is configured, verify that the kube-system pods are running.

      NOTE

      1. If the Kubernetes controller-manager is in CrashbackLoopOff, then it is likely failing to authenticate during bootstrap. Verify that the service-account-signing-key-file matches the service-account-key-file specified in the kube-controller-manager.yaml.
      2. If the service-account-issuer has extra trailing characters such as a forward slash (/), it fails authentication with Fortanix DSM.

      7.3  Signing Key Extraction for Fortanix DSM

      The public key corresponding to the signing key of the Kubernetes API Server is needed so that Fortanix DSM can verify the JWT upon receipt of an authentication request.

      -- service-account-signing-key-file=/path/to/k8s/pki/token-signing.key         

      Also, if the pub file is not present within the Kubernetes PKI folder, then run the following command to get the public key:

      openssl rsa -in /path/to/k8s/pki/token-signing.key -pubout > key.pub        

      7.4  Fortanix DSM Configuration For Auto-Auth

      1. Set up a group in your Fortanix DSM account for the purpose of storing the secrets.
      2. Create an app in the above Group with the following parameters:
         1. Name: Should be exactly as system:serviceaccount:default:<serviceaccountname>

            For example: system:serviceaccount:default:default (for default service account)

         2. Authentication Method: JSON Web Token.
         3. Valid Issuer(s): Should match the specification of Kubernetes API Server (Section 6.2).

            -- service-account-issuer=https://token-issuing.application.com         

         4. Upload the Public Key (and Key ID if exists) obtained from Section 7.3.
            The public key extracted in the previous step can also be dynamically fetched for verification. This supports auto-TLS if it is set up in Kubernetes, wherein the keys and certificates are refreshed more frequently.
      3. For Secrets injection use-case, import Secrets in the above Group, which are required by your Kubernetes applications.
         1. Import Security Object. Select type as “Secret”.
            NOTE

            Any type of security object can be injected and is not limited to Secrets.
         2. Use any name which is indicative of the secret. For example: “org-xyz-db-oracle-admin-password”.
         3. Import the secret value as RAW data type.
         4. Make sure to select “EXPORT” as one of the key operations.

      8.0  Annotations for Auto Secrets Injections

      8.1  Annotations for File-Based Secrets Injections

      This section describes the annotations required in application pod/deployment for secrets to be injected as files.

      • An emptyDir type Kubernetes volume is mounted by the init container to the application container(s) in a pod.
      • This volume hosts all the secrets injected as separate files. This mount path is configurable.
      • This mounted volume is owned by root:root and file permissions are set to 750. Non-root user or lesser permissions is not yet supported.

      The following example mounts two secrets on the pods, at the file locations:

      • /opt/myapp/credentials/xyzpassword.txt
      • /opt/myapp/credentials/abcpassword.txt

      apiVersion: v1
      kind: Pod
      metadata:
        namespace: demo-apps
        name: sample-pod
        annotations:
           secrets-injector.fortanix.com/secrets-volume-path: /opt/myapp/credentials
           secrets-injector.fortanix.com/inject-secret-xyzpassword.txt: "org-xyz-db-oracle-admin-password"
           secrets-injector.fortanix.com/inject-secret-abcpassword.txt: "org-abc-db-oracle-admin-password"
      spec:
         containers:
         - name: myapp
           image: myapp:1.0        

      Where
      • secrets-injector.fortanix.com/secrets-volume-path is the folder where all secrets are saved as files. If not set, the default location is  /var/run/fortanix/secrets in the target pod.
      • The annotation for specifying a Secret to be injected is:

        secrets-injector.fortanix.com/inject-secret-<secret-mount-file>: <SDKMS Security Object name>

      • <secret-mount-file> is the file name to be mounted in the above folder which contains the secret value upon injection.
      • <SDKMS Security Object name> is the name of Fortanix DSM Security Object which was imported in step 4.3.1.
      • Multiple such entries can be added for injecting multiple secrets.

8.2  Annotations for Environment Variable Based Secrets Injections

This section describes the annotations required in application pod/deployment for secrets to be injected as environment variables.

Two emptyDir type Kubernetes volumes are mounted by the init container to the application container(s) in a pod.

• Volume mounted at /var/run/fortanix/secrets hosts all secrets as separate files.
• Volume mounted at /opt/fortanix contains a script env-wrapper which is used to load these secrets as environment variables in the main process.

Hence this mode of secret injection is supported only if the pod/deployment spec explicitly defines the container command and do not rely of docker image’s default entry-point.

• The following example mounts two secrets as environment variables XYZ_DB_PASS and ABC_DB_PASS.

  apiVersion: v1
  kind: Pod
  metadata:
    namespace: demo-apps
    name: sample-pod
    annotations:
       secrets-injector.fortanix.com/inject-secret-XYZ_DB_PASS: "org-xyz-db-oracle-admin-password"
       secrets-injector.fortanix.com/inject-as-env-XYZ_DB_PASS: "true"
       secrets-injector.fortanix.com/inject-secret-ABC_DB_PASS: "org-abc-db-oracle-admin-password"
       secrets-injector.fortanix.com/inject-as-env-ABC_DB_PASS: "true"
  spec:
     containers:
     - name: myapp
       image: myapp:1.0
       command: 
       - /bin/myapp
       args:
       - "--flag=true"       

• The annotation for specifying a Secret to be injected is (there are two annotations per secret to be injected):
  • secrets-injector.fortanix.com/inject-secret-<secret-env-variable>: <SDKMS Security Object name>
  • secrets-injector.fortanix.com/inject-as-env-<secret-env-variable>: "true"
    • <secret-env-variable> is the environment variable that contains the secret value upon injection.
    • <SDKMS Security Object name> is the name of the Fortanix DSM Security Object that was imported in Section 6.4.

8.3  Annotations for Both Env Variable AND File Based Secrets Injections

This section describes the annotations required in application pod/deployment for multiple secrets to be injected as environment variables as well as files. The steps are combination of Section 8.1 and 8.2.

The following example will mount two secrets as environment variables XYZ_DB_PASS and ABC_DB_PASS and two more secrets xyzpassword.txt and abcpassword.txt as files mounted in a particular location.

apiVersion: v1
kind: Pod
metadata:
  namespace: demo-apps
  name: sample-pod
  annotations:
    secrets-injector.fortanix.com/secrets-volume-path: /opt/myapp/credentials
    secrets-injector.fortanix.com/inject-secret-XYZ_DB_PASS: "org-xyz-db-oracle-admin-password"
    secrets-injector.fortanix.com/inject-as-env-XYZ_DB_PASS: "true"
    secrets-injector.fortanix.com/inject-secret-ABC_DB_PASS: "org-abc-db-oracle-admin-password"
    secrets-injector.fortanix.com/inject-as-env-ABC_DB_PASS: "true"
    secrets-injector.fortanix.com/inject-secret-xyzpassword.txt: "org-xyz-db-oracle-admin-password"
    secrets-injector.fortanix.com/inject-secret-abcpassword.txt: "org-abc-db-oracle-admin-password"
spec:
  containers:
  - name: myapp
    image: myapp:1.0
    command: 
    - /bin/myapp
    args:
    - "--flag=true"         

9.0 Conclusion

DevOps is fast transitioning into “DevSecOps”. Security teams are tasked during application development to model network topology as well as to protect the credentials and secrets used to provision infrastructure, deploy applications, and/or databases. Passwords and API tokens are constantly spread out within the enterprise and often stored on a disk using configuration files or data bags. Security typically falls to dedicated roles but involves all other teams that are delivering an application.

Kubernetes Service Account Token Projection in Amazon Web Services (AWS) is used to tailor a fine-grained RBAC (role-based-access-control) mechanism in conjunction with the AWS IAM (identify-and-authorization-management) for its EKS offering.

9.1  Other Examples

More details are available at
kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection

AWS guide on the provisioning of fine-grained IAM roles for Kubernetes service accounts: aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/