Using Fortanix Data Security Manager for Secrets Injection in OpenShift

1.0 Introduction

This article describes how to integrate Fortanix-Data-Security-Manager (DSM) with OpenShift for managing and securing application secrets.

It also contains the information that a user requires to: 

• Encrypt OpenShift secrets in etcd using Fortanix DSM OpenShift KMS Plugin.

• Install Fortanix DSM secrets injection controller in  OpenShift cluster for automatic secrets injection at runtime to applications.

• Set up controller to authenticate using Fortanix DSM Application Processing Interface (API) Keys.

• Set up controller to authenticate using OpenShift Service Account JSON Web Tokens (JWTs).

• Deploy an application with auto secrets injection enabled.

2.0 Background

Cloud-native and hybrid applications are increasingly deployed within containers. OpenShift is transforming not only the application architecture but also the development and operations pipelines for application delivery.

Every day, teams come to  rely on numerous secrets in  their development and operational DevOps processes. Secrets ranging from passwords, tokens, certificates, Secure Shell (SSH) keys, and database credentials simply cannot  be hardcoded or configured statically anymore.

Fortanix offers a powerful yet straightforward solution for enterprises and DevOps  alike  to centralize the storage and management of security artifacts. Secrets, cryptographic keys, and certificates can be extended to widely used applications such as  databases, Public Key Infrastructure (PKI) systems, HyperText Transfer Protocol (HTTP) servers/proxies, and custom applications. Most importantly, it does so  without sacrificing  the security of a Hardware Security Module (HSM) and  the agility of a massively scalable Google Cloud Key Management Service (KMS) built on a modern technology stack.

2.1 Secrets Management in OpenShift

OpenShift operates in two primary realms: the control plane and data plane control plane and data plane. Various components within the control plane are responsible for the orchestration of backbone services necessary to run a distributed application.

OpenShift relies on internal secrets to bootstrap  various critical services in the control plane to establish  trust. These secrets are stored in etcd database and are only base64-encoded by default. Fortanix DSM can encrypt the entire  etcd data on behalf of OpenShift to ensure data and application security at-rest. This is a critical requirement for operating any enterprise OpenShift cluster. Encryption keys do not leave the HSM, as OpenShift invokes Fortanix DSM to encrypt or decrypt the secondary data encryption keys  using a master key. This is accomplished  using the OpenShift KMS Plugin as described in Section 3.0: OpenShift KMS Plugin.

In addition to serving as an external KMS provider to OpenShift, Fortanix DSM can also directly own and manage application-level secrets for OpenShift. The secrets are never stored at rest in local databases such as etcd, instead, they are injected directly into the pods at runtime. This document  also illustrates on multiple ways of authentication methods for accessing secrets in Fortanix DSM. For more information, refer to Section 4.0: OpenShift Secrets Injection - Architecture.

Fortanix DSM supports multiple authentication mechanisms. API Key is the standard method  for authentication and is stored as a OpenShift secret within the cluster . Additionally, OpenShift and Google Cloud service accounts  can authenticate using a trustworthy, OpenShift-generated, signed JWT. This mechanism maintains the credentials independently from the application simply by relying on the PKI established within the orchestration control plane, thereby enhancing security.

As many enterprises find their DevOps workflows to the Cloud, and with OpenShift increasingly provisioning a variety of distributed deployments, the need for robust application secrets stored as a supplement to OpenShift etcd should be paramount.

3.0 OpenShift KMS Plugin

Effective OpenShift 1.10, a new feature called KMS Encryption Provider lets organizations bring their own KMS, for encrypting secrets stored in OpenShift etcd database. Fortanix provides an open-source plugin that communicates with OpenShift over a Unix Socket. At bootup or during orchestration, the API Server automatically generates the DEKs and encrypts them using a master key (KEK) stored in Fortanix DSM.

3.1 Prerequisites

Ensure that the following requirements are met:

• OpenShift cluster, Version 1.10 and above.

• Support for OpenShift 4.1 and above.

• Access to OpenShift Master nodes is required.

3.2 Setup

To encrypt the OpenShift secrets with Key(s) stored in Fortanix DSM, refer to Fortanix GitHub Repository - Readme.

To deploy the OpenShift cluster using Rancher, refer to Fortanix GitHub Repository – Rancher Guide to apply the encryption configuration.

4.0 OpenShift Secrets Injection - Architecture

Instead of storing sensitive data as OpenShift secrets, secrets can be directly injected into  workloads at runtime using Admission Controllers provided by Fortanix. This provides tighter control over  sensitive data, ensuring that secrets are not stored in any local database or file system at rest. Developers and operators only need to specify the application secrets, which are automatically injected into destination pods or containers as environment variables.

Admission Controllers were introduced in OpenShift version 1.9 and made publicly available (beta) since version 1.13. They are commonly implemented as Validation or Mutation Webhooks that inspect, verify and/or alter pod specifications during their creation or update within the cluster. When implemented with a dedicated sidecar for reading and refreshing secrets from Fortanix DSM, secrets are securely injected into the application. A cluster administrator  updates the API Server to accept Admission Controllers. A Mutating Webhook Configuration is applied, which runs a OpenShift Service to inspect pod specifications and can be limited to either the default or a specific namespace. This mutation service is a simple web server that leverages the OpenShift API to read the config maps, volumes and, containers of the pod seeking entry into or update within the cluster. Pod specifications must include certain predefined annotations relevant to the mutation service. Optionally, the mutation service can be programmed to perform dynamic changes to the pod based on a flexible set of annotations.

This mutation service is a simple Web Server that is implemented in Go programming language supporting the OpenShift API.

This web server typically does the following:

1. A main method to keep the webserver daemon running.

2. The HTTP endpoint at which requests from API Server are received, for example: /mutate

3. Transport Layer Security (TLS) certificates for mutual authentication and secure exchange of payload.

The /mutate request typically invokes another package or function that performs a OpenShift API Admission Review to:

1. Validate if the request is compatible with the configuration

   • pod namespace

   • pod annotations

   • pod associated service account

2. Mutate the request and dispatch the response such that the pod specification includes:

   • Config maps for any variables needed by the sidecar containers

     • Environment variables

   • Volumes

     • Projected Service Account token if using JWT-based authentication

   • Secrets, if needed

   • Sidecar Containers

     • Init Container

     • Refresh Container (optional)

   • Update pod annotations to reflect that:

     • Mutation or secrets injection is applied

     • Additional parameters have been passed to the sidecar or main app containers

Now, every time an application deployment or replica sets attempts to create or update pods in the intended or target namespace, the Mutating Webhook intercepts and injects the OpenShift artifacts (config maps, volumes, and, most importantly the sidecar containers). Since Fortanix DSM is a vault-less and agent-less KMS and Secrets repository, the init and refresh containers are simple commands instead of elaborate service daemons or processes. These containers authenticate with Fortanix DSM using either an API key or a trustworthy JWT generated by OpenShift for the pod. Once authenticated, the sidecar containers retrieve secrets based on role-based access control (RBAC)  configured within Fortanix DSM,  Fortanix DSM identified by the API KEY or OpenShift service account . The secrets  are then injected directly into the pod and provisioned as environment variables. Ultimately, this allows applications to remain unaltered and their dependency on secrets seamlessly switches to Fortanix DSM from legacy repositories.

5.0  Secrets Injection Admission Controller

5.1  Prerequisites

Ensure that the following requirements are met:

• KuOpenShift bernetes cluster, Version 1.16 and above.

• Also supports OpenShift 4.3 and above.

• Fortanix DSM Account

  • Either on the Cloud or locally within your enterprise HSM.

• Helm Version 3 and above.

  • Required for secrets injection controller installation.

5.2  Setup

5.2.1  OpenShift (Cloud and On-premises)

1. Make sure the kube-config is set up to access your cluster.

2. Download the Fortanix DSM admission controller helm chart from here.

3. Extract the chart.

   tar -xf dsm-secrets-injector-chart-x.x.tar.gz

   This creates a folder called dsm-secrets-injector-chart containing the helm chart details.

4. Install the admission controller chart:

   helm install dsm-secrets-injector ./dsm-secrets-injector-chart

   The output may resemble as follows:

   NAME: dsm-secrets-injector
   LAST DEPLOYED: Thu Jul 1 10:34:58 2021
   NAMESPACE: default
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   NOTES:
   OpenShift Integration with Fortanix DSM has been deployed successfully.

5. Verify if admission controller pods are healthy:

   kubectl get pods --namespace fortanix
   

   The output may resemble as follows:

   NAME                                         READY   STATUS      RESTARTS   AGE
   fortanix-cert-setup-5g92h                    0/1     Completed   0          4h21m
   fortanix-secrets-injector-59cc799797-gx9tq   1/1     Running     0          4h21m
   kubectl logs fortanix-secrets-injector-59cc799797-gx9tq --namespace fortanix
   2021/01/29 03:45:46 Configuration:
   2021/01/29 03:45:46   controllerConfigFile: /opt/fortanix/controller-config.yaml
   2021/01/29 03:45:46   webServerConfig.port: 8443
   2021/01/29 03:45:46   webServerConfig.certFile: /opt/fortanix/certs/cert.pem
   2021/01/29 03:45:46   webServerConfig.keyFile: /opt/fortanix/certs/key.pem
   2021/01/29 03:45:46 AuthTokenType: api-key
   2021/01/29 03:45:46 SecretAgentImage: registry.hub.docker.com/fortanix/k8s-sdkms-secret-agent:1.1

6. Run the following command to view the logs of the Fortanix Admission Webhook:

   kubectl logs dsm-secrets-injector-chart-<variable_number> --namespace fortanix
   

   For example:

   kubectl logs dsm-secrets-injector-chart-59cc799797-gx9tq --namespace fortanix
   2021/01/29 03:45:46 Configuration:
   2021/01/29 03:45:46   controllerConfigFile: /opt/fortanix/controller-config.yaml
   2021/01/29 03:45:46   webServerConfig.port: 8443
   2021/01/29 03:45:46   webServerConfig.certFile: /opt/fortanix/certs/cert.pem
   2021/01/29 03:45:46   webServerConfig.keyFile: /opt/fortanix/certs/key.pem
   2021/01/29 03:45:46 AuthTokenType: api-key
   2021/01/29 03:45:46 SecretAgentImage: registry.hub.docker.com/fortanix/k8s-sdkms-secret-agent:1.1

5.2.2 Openshift

1. Make sure crc and oc is set up to access your cluster. The following command should successfully return the cluster information:

   oc cluster-info 

2. Make sure helm is installed.

3. Download the Fortanix DSM admission controller helm chart from here.

4. Extract the chart:

   tar -xf dsm-secrets-injector-chart-x.x-openshift.tar.gz

   This creates a folder called dsm-secrets-injector-chart containing the helm chart details.

5. Install the admission controller chart:

   helm install dsm-secrets-injector ./dsm-secrets-injector-chart

   The output may resemble as follows:

   NAME: dsm-secrets-injection
   LAST DEPLOYED: Thu Jul 1 10:34:58 2021
   NAMESPACE: default
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   NOTES: OpenShift Integration with Fortanix DSM has been deployed successfully.

6. Run the following command to verify if the admission controller pods are healthy:

   oc project fortanix

   The output may resemble as follows:

   Now using project "fortanix" on server "https://api.crc.testing:6443".

7. Run the following command to check the status of the pods:

   oc status
   

   The output may resemble as follows:

   In project fortanix on server https://api.crc.testing:6443
   svc/fortanix-secrets-injector-svc - 10.217.4.34:443 -> 8443 
   deployment/fortanix-secrets-injector deploys fortanix/k8s-sdkms-secrets-injector:1.0
   deployment 1 running for 4 minutes - 1 pod
   job/fortanix-cert-setup manages fortanix/k8s-sdkms-cert-setup:1.0
   created 4 minutes ago 1/1 completed 0 running
   2 infos identified, use 'oc status --suggest' to see details.
   

8. Run the following command to view the list of the pods:

   oc get pods

   The output may resemble as follows:

   secrets-injector-cert-setup-5g92h                    0/1     Completed   0          4h21m
   fortanix-secrets-injector-59cc799797-gx9tq   1/1     Running     0          4h21m      

9. Run the following command to view the list of the logs of the Fortanix Admission Webhook:

   oc logs fortanix-secrets-injector-59cc799797-gx9tq
   

   For example:

   oc logs fortanix-secrets-injector-59cc799797-gx9tq
   2021/01/29 03:45:46 Configuration:
   2021/01/29 03:45:46 controllerConfigFile: /opt/fortanix/controller-config.yaml
   2021/01/29 03:45:46 webServerConfig.port: 8443
   2021/01/29 03:45:46 webServerConfig.certFile: /opt/fortanix/certs/cert.pem
   2021/01/29 03:45:46 webServerConfig.keyFile: /opt/fortanix/certs/key.pem
   2021/01/29 03:45:46 AuthTokenType: api-key
   2021/01/29 03:45:46 SecretAgentImage: registry.hub.docker.com/fortanix/k8s-sdkms-secret-agent:ubi7-2.3
   

5.3  Troubleshooting and Maintenance

The Admission Controller runs as a standalone webservice, with the sole purpose of mutating pod specifications served OpenShift API server. Please note the following aspects of the webservice:

1. It never communicates with Fortanix DSM or any other external entity.

2. It never handles any of the  secrets directly. The controller  only patches pods with information about  the Fortanix DSM init container and metadata for the secrets to be injected. The actual  secrets are managed by init containers only.

3. Failure of the Admission Controller  webservice affects only new pod creation or restarts. Existing pods, which have secrets already injected, are not affected.

5.3.1  Multiple Replicas

It is advisable to run multiple instances of controller in production, for better resiliency. Do the following while installing the helm chart:

helm install dsm-secrets-injector ./dsm-secrets-injector-chart --set injector.replicas=3

5.3.2  Web Service TLS Expiration

The admission controller is a webservice over TLS. By default, it uses a self-signed certificate with a validity of 1 year. One can change the validity if needed as follows during installation.

helm install dsm-secrets-injector ./dsm-secrets-injector-chart --set injector.certExpiration=730
//if needed to be keep for 2 years

5.3.3  Upgrade

If you wish to upgrade the controller to a newer released version, download the new helm chart and run the following helm upgrade command:

helm upgrade dsm-secrets-injector ./dsm-secrets-injector-chart-new-version

6.0 Configure Fortanix DSM

A Fortanix DSM service must be configured, and the URL must be accessible. To create a Fortanix DSM account and group, refer to the following sections:

6.1 Signing Up

6.2 Creating an Account

Access <Your_DSM_Service_URL> in a web browser and enter your credentials to log in to Fortanix DSM.

6.3 Creating a Group

Perform the following steps to create a group in the Fortanix DSM:

1. In the DSM left navigation panel, click the Groups menu item, and then click ADD GROUP to create a new group.

   

2. On the Adding new group page, do the following:

   1. Title: Enter a name for your group.

   2. Description (optional): Enter a short description of the group.

3. Click SAVE to create the new group.

The new group is added to the Fortanix DSM successfully.

6.4 Creating an Application

Perform the following steps to create an application (app) in the Fortanix DSM:

1. In the DSM left navigation panel, click the Apps menu item, and then click ADD APP to create a new app.

   

2. On the Adding new app page, do the following:

   1. App name: Enter the name for your application.

   2. ADD DESCRIPTION (optional): Enter a short description of the application.

   3. Authentication method: Select the default API Key as the authentication method from the drop down menu. For more information on these authentication methods, refer to the User's Guide: Authentication.

   4. Assigning the new app to groups: Select the group created in Section 6.3: Creating a Group from the list.

3. Click SAVE to add the new application.

The new application is added to the Fortanix DSM successfully.

6.5 Copying the App Key

Perform the following steps to copy the app UUID from the Fortanix DSM:

1. In the DSM left navigation panel, click the Apps menu item, and then click the app created in Section 6.4: Creating an Application to go to the detailed view of the app.

2. On the INFO tab, click VIEW API KEY DETAILS.

3. From the API Key Details dialog box, copy the API Key of the app to use it later.

6.6 Importing a Security Object

Perform the following steps to import security objects in the Fortanix DSM:

1. In the DSM left navigation panel, click the Security Objects menu item, and then click ADD SECURITY OBJECT to create a new security object.

   

2. On the Add new Security Object page, do the following:

   1. Security Object Name: Enter a name for your security object.

      NOTE

      Use any name which is indicative of the secret. For example, org-xyz-db-oracle-admin-password.

   2. Group: Select the group as created in Section 6.3: Creating a Group.

   3. Select the IMPORT radio button.

   4. In the Choose a type section, select the RSA key type.

      NOTE

      Any type of security object  can be injected and is not limited to Secrets.

   5. In the Place value here or import from file section, select the value format type as Raw only and click the UPLOAD A FILE button to upload the key file.

   6. In the Key operations permitted section, select the required operations to define the actions that can be performed with the cryptographic keys, such as encryption, decryption, signing, and verifying, except EXPORT.

3. Click IMPORT to create the new security object.

7.0  Secrets Injection (API-based authentication)

Perform the followings steps to enable auto-injection of secrets in OpenShift application, following modifications are required in the existing application setup:

1. Update the OpenShift namespace/OpenShift project used by the application, to add a new label fortanix-secrets-injector: enabled. This enables the namespace/project for automated secrets injection.

   For example, if your applications are running in a namespace called demo-apps (This can be the default namespace too), edit the namespace as follows:

    OpenShift:

   oc label namespace demo-apps fortanix-secrets-injector=enabled

2. Add a secret in the demo-apps namespace which stores the Fortanix DSM authentication parameters. The following are the required parameters:

   1. api_endpoint: API Endpoint is the same as your DSM deployment endpoint. For example: https://smartkey.io/

   2. api_key: API Key is the same as copied from Fortanix DSM Dashboard.

   3. name: The name of the secret is prefixed by  serviceAccount of the target application. In the following example, demo-sa is the serviceAccount used by the application, hence the name is demo-sa-sdkms-credentials. If the serviceAccount is not set explicitly by the application, then use the value default (the name of the secret would then be default-sdkms-credentials)

   4. cacert: Fortanix DSM CA Certificate. This is required only if the DSM endpoint is served using a custom organization CA. (Optional).

   5. The secret needs to be created separately for each namespace and serviceAccount used by the applications.          

      OpenShift:

      $ oc create secret generic
      demo-sa-sdkms-credentials --from-file=cacert=dsm-ca.crt
      --from-literal api_endpoint=https://smartkey.io/
      --from-literal api_key=NCNNJIEDN....03ninCWW0i0n
      -n demo-apps        

3. Update your application pod/deployment/statefulset/daemonset specification to add annotations for enabling secrets injection. One can inject secrets as:

   1. Environment Variables

   2. Files mounted at a particular location.

   3. Both env variables and mounted files.

   4. Multiple secrets can be mounted at the same time.

   5. You will need to restart existing pods for annotations to take effect. Once added, any restart of the deployment/pods will automatically inject the secrets.

   6. Refer to Section 9.0: Annotations for Auto Secrets Injections on how to update application annotations for auto secrets injection.

8.0 Secrets Injection (Auto-Auth using Trustworthy Tokens)

OpenShift 1.12 released a new feature called Service Account Token Volume Projection.

This allows applications to rely on named service accounts and add an  additional layer of security for authentication and authorization. This also addresses the “Secret Zero” problem by using OpenShift provided JWT tokens for authentication (instead of API Key).

Upon deployment, the API Server dynamically injects the service account identity into the pod(s) during runtime to provide an attestation. The identity provides a verifiable way for external applications or third parties to inspect a signed token and grant  additional permissions to resources. Applications running within separate containers inside the same pod will have access to the signed token. The signing of identity tokens is configurable by the cluster administrator by patching the API Server.

When Fortanix DSM receives and verifies the token using the public key of the OpenShift API server , it can validate additional roles and access controls for its native resources. Since these signed tokens have a finite expiration period and are rotated automatically by OpenShift based on the pod spec ification, the risk associated with capturing a single token is minimal, thereby significantly reducing the threat surface.

Since the signed token is generated from the service account, which itself is encrypted at rest, the JWT is never stored during injection by OpenShift into the pod.

8.1  Security Principles

Each service account in OpenShift gets a default token. These tokens are signed by the API server , but not rotated and have unlimited validity during their existence. Using these tokens is perhaps equivalent to using a legacy password. OpenShift took note and created the token volume projection to address the security gaps. These signed tokens are rotated automatically based on pod specifications and their validity is limited in scope based on attributes outlined below:

• Associated with a single OpenShift service account.

• Permissioned by the cluster role and role bindings.

• Identified by an issuer name or URL.

• Distributed to a specific audience for whom the token is generated and targeted.

• Invalidated outside of its time window through a Not-Before or Valid-After Time to Live (TTL).

• Signed by the API Server private key and verifiable by its public key.

Based on these tenets, the signed tokens (JWT) are more trustworthy than standard service account tokens and when automatically authorized by Fortanix DSM, render applications access to their secrets, keys, or certificates seamlessly. This tremendously improves the authentication of applications, services and/or machines by removing the need to manage individual identities.

8.2  Kube API Server Update for Signing Key

Depending upon the OpenShift cluster, this step can require a primitive configuration change in the API Server (/etc/OpenShift/manifest/kube-apiserver.yaml).

• “Audience” value should be the Fortanix DSM endpoint URL.

• “Issuer” to denote your cluster/application. This is used to identify the tokens for this cluster when pods authenticate with Fortanix DSM for secrets injection.

• “Signing Key File” is the signing key selected for token generation. Fortanix DSM uses the public part of this key to verify the tokens. One can create a new signing key using openssl as:

  openssl genrsa -out token-signing.key 2048

8.2.1 Minikube

Start Minikube with these additional flags to update the API Server:

minikube start \
   --extra-config=apiserver.service-account-issuer=https://k8s.test \
   --extra-config=apiserver.service-account-api-audiences=https://smartkey.io \
   --extra-config=apiserver.service-account-signing-key-file=/path/to/k8s/pki/token-signing.key

8.2.2 Docker Desktop

Docker Desktop requires editing kube-apiserver.yaml manifest by accessing the master or host node:

--service-account-issuer=https://token-issuing.application.com
--service-account-api-audiences=https://smartkey.io
--service-account-signing-key-file=/path/to/k8s/pki/token-signing.key

Here is a Docker Desktop command to get remote access to the master node:

docker run --rm -it --privileged --pid=host walkerlee/nsenter \
-t 1 -m -- vi /etc/OpenShift/manifests/kube-apiserver.yaml        

Once the API server is configured, verify that the kube-system pods are running.

8.3  Signing Key Extraction for Fortanix DSM

The public key corresponding to the signing key of the OpenShift API Server is needed so that Fortanix DSM can verify the JWT upon receipt of an authentication request.

-- service-account-signing-key-file=/path/to/k8s/pki/token-signing.key         

Also, if the pub file is not present within the OpenShift PKI folder, then run the following command to get the public key:

openssl rsa -in /path/to/k8s/pki/token-signing.key -pubout > key.pub        

8.4  Fortanix DSM Configuration For Auto-Auth

1. Set up a group in your Fortanix DSM account to store the secrets.

2. Create an app in the above Group with the following parameters:

   1. Name: Should be exactly as system:serviceaccount:default:<serviceaccountname>

      For example: system:serviceaccount:default:default (for default service account)

   2. Authentication Method: JSON Web Token.

   3. Valid Issuer(s): Should match the specification of OpenShift API Server (Section 8.2: Kube API Server Update for Signing Key).

      -- service-account-issuer=https://token-issuing.application.com         

   4. Upload the Public Key (and Key ID if exists) obtained from Section 8.3: Signing Key Extraction for Fortanix DSM.
      The public key extracted in the previous step can also be dynamically fetched for verification. This supports auto-TLS if it is set up in OpenShift, wherein the keys and certificates are refreshed more frequently.

3. For the Secrets injection use-case, import Secrets in the above Group, which are required by your OpenShift applications.

   1. Import Security Object. Select the type as “Secret”.

      NOTE

      Any type of security object can be injected and is not limited to Secrets.

   2. Use any name that is indicative of the secret. For example: org-xyz-db-oracle-admin-password.

   3. Import the secret value as RAW data type.

   4. Make sure to select “EXPORT” as one of the key operations.

9.0  Annotations for Auto Secrets Injections

9.1  Annotations for File-Based Secrets Injections

This section describes the annotations required in the application pod/deployment for secrets to be injected as files.

• An emptyDir type OpenShift volume is mounted by the init container to the application container(s) in a pod.

• This volume hosts all the secrets injected as separate files. This mount path is configurable.

• This mounted volume is owned by root:root and file permissions are set to 750. Non-root user or lesser permissions is not yet supported.

The following example mounts two secrets on the pods, at the file locations:

• /opt/myapp/credentials/xyzpassword.txt

• /opt/myapp/credentials/abcpassword.txt

apiVersion: v1
kind: Pod
metadata:
  namespace: demo-apps
  name: sample-pod
  annotations:
     secrets-injector.fortanix.com/secrets-volume-path: /opt/myapp/credentials
     secrets-injector.fortanix.com/inject-secret-xyzpassword.txt: "org-xyz-db-oracle-admin-password"
     secrets-injector.fortanix.com/inject-secret-abcpassword.txt: "org-abc-db-oracle-admin-password"
spec:
   containers:
   - name: myapp
     image: myapp:1.0        

Where

• secrets-injector.fortanix.com/secrets-volume-path is the folder where all secrets are saved as files. If not set, the default location is  /var/run/fortanix/secrets in the target pod.

• The annotation for specifying a Secret to be injected is:

  secrets-injector.fortanix.com/inject-secret-:-<secret-mount-file>: <SDKMS Security Object name>

• <secret-mount-file> is the file name to be mounted in the above folder which contains the secret value upon injection.

• <SDKMS Security Object name> is the name of Fortanix DSM Security Object which was imported in Section 6.0: Configure Fortanix DSM.

• Multiple such entries can be added for injecting multiple secrets.

9.2  Annotations for Environment Variable-Based Secrets Injections

This section describes the annotations required in the application pod/deployment for secrets to be injected as environment variables.

Two emptyDir type OpenShift volumes are mounted by the init container to the application container(s) in a pod.

• Volume mounted at /var/run/fortanix/secrets hosts all secrets as separate files.

• Volume mounted at /opt/fortanix contains a script env-wrapper that is used to load these secrets as environment variables in the main process.

Hence this mode of secret injection is supported only if the pod/deployment spec explicitly defines the container command and do not rely on the docker image’s default entry point.

The following example mounts two secrets as environment variables XYZ_DB_PASS and ABC_DB_PASS.

apiVersion: v1
kind: Pod
metadata:
  namespace: demo-apps
  name: sample-pod
  annotations:
     secrets-injector.fortanix.com/inject-secret-XYZ_DB_PASS: "org-xyz-db-oracle-admin-password"
     secrets-injector.fortanix.com/inject-as-env-XYZ_DB_PASS: "true"
     secrets-injector.fortanix.com/inject-secret-ABC_DB_PASS: "org-abc-db-oracle-admin-password"
     secrets-injector.fortanix.com/inject-as-env-ABC_DB_PASS: "true"
spec:
   containers:
   - name: myapp
     image: myapp:1.0
     command: 
     - /bin/myapp
     args:
     - "--flag=true"       

The annotation for specifying a Secret to be injected is (there are two annotations per secret to be injected):

• secrets-injector.fortanix.com/inject-secret-:<secret-env-variable>: <SDKMS Security Object name>

• secrets-injector.fortanix.com/inject-as-env-<secret-env-variable> "true"

Where,

• <secret-env-variable> is the environment variable that contains the secret value upon injection.

• <SDKMS Security Object name> is the name of the Fortanix DSM Security Object that was imported in Section 6.1: Configure Fortanix DSM.

9.3  Annotations for Both Env Variable AND File-Based Secrets Injections

This section describes the annotations required in application pod/deployment for multiple secrets to be injected as environment variables as well as files. The steps are a combination of Sections 9.1: Annotations for File-Based Secrets Injections and Section 9.2: Annotations for Environment Variable-Based Secrets Injections.

The following example will mount two secrets as environment variables XYZ_DB_PASS and ABC_DB_PASS and two more secrets xyzpassword.txt and abcpassword.txt as files mounted in a particular location.

apiVersion: v1
kind: Pod
metadata:
  namespace: demo-apps
  name: sample-pod
  annotations:
    secrets-injector.fortanix.com/secrets-volume-path: /opt/myapp/credentials
    secrets-injector.fortanix.com/inject-secret-XYZ_DB_PASS: "org-xyz-db-oracle-admin-password"
    secrets-injector.fortanix.com/inject-as-env-XYZ_DB_PASS: "true"
    secrets-injector.fortanix.com/inject-secret-ABC_DB_PASS: "org-abc-db-oracle-admin-password"
    secrets-injector.fortanix.com/inject-as-env-ABC_DB_PASS: "true"
    secrets-injector.fortanix.com/inject-secret-xyzpassword.txt: "org-xyz-db-oracle-admin-password"
    secrets-injector.fortanix.com/inject-secret-abcpassword.txt: "org-abc-db-oracle-admin-password"
spec:
  containers:
  - name: myapp
    image: myapp:1.0
    command: 
    - /bin/myapp
    args:
    - "--flag=true"         

10.0 Conclusion

DevOps is fast transitioning into “DevSecOps”. Security teams are tasked during application development to model network topology as well as to protect the credentials and secrets used to provision infrastructure, deploy applications, and/or databases. Passwords and API tokens are constantly scattered across the enterprise  within the enterprise and often stored on a disk using configuration files or data bags. Security typically falls to dedicated roles but involves all other teams that are delivering an application.

OpenShift Service Account Token Projection in Amazon Web Services (AWS) is used to tailor a fine-grained RBAC (role-based-access-control) mechanism in conjunction with the AWS IAM (identify-and-authorization-management) for its EKS offering.

10.1  Other Examples

For more information, refer to kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection.

For more information about AWS provisioning of fine-grained IAM roles for Kubernetes service accounts, refer to  aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/.