CMK Rotation - Fortanix DSM Version 5.6 and Above

1.0 Introduction

The purpose of this article is to describe the process of rotating the Cluster Master Key (CMK) for Fortanix-Data-Security-Manager (DSM) cluster running version 5.6 and above.

2.0 Cluster Master Key Rotation – DSM Version 5.6 and Above

Fortanix DSM uses a CMK to derive individual keys for functions such as database encryption.

The following section describes the steps to rotate the CMK using shell and kubectl on one of the Fortanix DSM nodes:

2.1 Create a New External Deployment Key

This section creates a new external Deployment Key in the External HSM (Fortanix DSM), which will be used to derive the new CMK wrapping key after the CMK rotation.

Create a new AES 256 key in Fortanix DSM.
NOTE
Ensure that the key has “Export” permissions.
Figure 1: Create new AES 256 key - Deployment Key

2.2 Configure External HSM in DKS

This section configures the DKS to use deployment keys generated and stored in an external HSM. Once configured, the DKS will source deployment key material from the external HSM instead of using locally generated deployment keys.

Prepare the external HSM configuration file, ext_hsm.yaml with the extKeyId of the new key created in Section 2.1: Create a New External Deployment Key:
```
extHsm:
  kind: "Fortanix"
  url: "sit.smartkey.io"
  tls:
    mode: "required"
    validate_hostname: true
    ca:
      ca_set: "global_roots"
  pin: "<API key or HSM PIN>"
extKeyId:
  id: 157cd33d-1006-4843-9f66-e0bc45857939
```
Where,
- extHsm: Refers to the external HSM connection configuration.
- extKeyId.id: Refers to the identifier of the external deployment key to be used by the DKS for CMK wrapping key derivation.
Run the following command to patch the Kubernetes secret sdkms-deployment-key-store with the configuration provided in the above YAML file:
```
sudo sdkms-cluster rotate-dk --ext-hsm ext_hsm.yaml
```
This updates the Kubernetes secret to the following:
- Adds the external key reference into key_map.
- Stores the hmg_config.
- Adds the migration_svn.

Verify using the following command:

sudo kubectl get secret sdkms-deployment-key-store -o jsonpath='{.data}'
{"deployment_key_store":"eyJsYXRlc3Rfc3ZuIjoiQUFBQUFBQUFBQUFBQUFBQUFBQUFBQT09Iiwia2V5X21hcCI6eyJBQUFBQUFBQUFBQUFBQUFBQUFBQUFBPT0iOiJNV0JRcGczRkg5SFJ5dk1QbG1vSVVRPT0iLCJBUUVCQVFFQkFRRUJBUUVCQVFFQkFRPT0iOnsia2V5X2lkIjp7ImlkIjoiMTU3Y2QzM2QtMTAwNi00ODQzLTlmNjYtZTBiYzQ1ODU3OTM5In19fSwiaG1nX2NvbmZpZyI6eyJraW5kIjoiRm9ydGFuaXgiLCJ1cmwiOiJzaXQuc21hcnRrZXkuaW8iLCJ0bHMiOnsibW9kZSI6InJlcXVpcmVkIiwidmFsaWRhdGVfaG9zdG5hbWUiOnRydWUsImNhIjp7ImNhX3NldCI6Imdsb2JhbF9yb290cyJ9LCJjbGllbnRfa2V5IjpudWxsLCJjbGllbnRfY2VydCI6bnVsbH0sInBpbiI6Ik9ETmlZbUZsTVRjdE1qQXdNaTAwWmpNM0xXRmxNVFl0Wm1Oa01qWXhZakF4WW1GbE9tWllNR2hvWkZwVVpHSlZjemMyTW1RMFZ6TndUbG8zWkVGamIyRnlXa3hoTVdaRWJGOVFUbkZ3UkRFNGNrSlJTbmxTT0dWMFUwRjNVRU5mWjBOUFUzTlJSM2xUTjJGMVpGRndTRTFRYnpkWlRGWnRTRGxCIn0sIm1pZ3JhdGlvbl9zdm4iOiJBUUVCQVFFQkFRRUJBUUVCQVFFQkFRPT0ifQ=="}

$ echo '<above_value>' | base64 -d
{"latest_svn":"AAAAAAAAAAAAAAAAAAAAAA==",
 "key_map":{"AAAAAAAAAAAAAAAAAAAAAA==":"MWBQpg3FH9HRyvMPlmoIUQ==",
            "AQEBAQEBAQEBAQEBAQEBAQ==":{"key_id":
            {"id":"157cd33d-1006-4843-9f66-e0bc45857939"}}},
 "hmg_config":{"kind":"Fortanix","url":"sit.smartkey.io","tls":{"mode":"required","validate_hostname":true,"ca":{"ca_set":"global_roots"},"client_key":null,"client_cert":null},"pin":"ODNiYmFlMTct...SDlB"},
"migration_svn":"AQEBAQEBAQEBAQEBAQEBAQ=="}

Run the following command to navigate to the /bin directory and perform a rolling restart to load the updated DKS configuration:
```
cd /opt/fortanix/sdkms/bin
sudo ./dsm_backend_rolling_restart.sh
```
The pod logs show that the latest sdkms-deployment-key-store configuration is picked up.

2.3 Propose a New Master Key

This step describes the steps to rotate the CMK so that it is re-wrapped using a Platform Key derived from the newly configured external DKS. The CMK derivation remains local; however, its at-rest protection mechanism is updated to use deployment keys backed by the external HSM.

CMK rotation is required to complete the migration, as existing CMKs remain protected under the previous configuration until explicitly rotated.

Get the pod IPs for the sdkms pods using the following command:
```
sudo -E kubectl get pods -lapp=sdkms -owide
```
Propose a new CMK by making a REST API call to any one of the sdkms pods:
```
curl -kv https://<pod-IP>:4444/cluster/v1/key/propose -X POST
```
This should return 204 No Content.
Verify that the key_id has been fetched once using the Activity Logs in the detailed view of the Deployment Key in the Fortanix DSM user interface (UI).

**Figure 3: Deployment key activity log**

This shows that the external Deployment Key was fetched using the hmg_config to derive the CMK wrapping_key.

2.4 Distribute the Proposed Cluster Master Key

Distribute the new CMK to the rest of the sdkms pods by making a REST API call to each of the other sdkms pods:

curl -kv https://<other-pod-IP>:4444/cluster/v1/key/fetch -X POST -d '{"target":"<pod-IP>:4444"}'

Where,

pod-IP: Refers to the IP of the pod which received the propose API call.
other-pod-IP: Refers to the IP of the rest of the other sdkms pods.

This should return 204 No Content.

2.5 Verify the New Cluster Master Key

For each sdkms pod, get the cluster master key status:

curl -k https://<pod-IP>:4444/cluster/v1/key

Expected output:

{"cluster_key_version":1,"cluster_proposal_hash":"gJmhkQZvMEkn5VbyzBUSB9bcW61MhVlkBaiRjFuC/XA=","fetched_proposal_hash":"gJmhkQZvMEkn5VbyzBUSB9bcW61MhVlkBaiRjFuC/XA="}

NOTE
The cluster_key_version is still 1 which means that the existing old CMK is still applicable.

2.6 Restart the `sdkms` Pods

Restart all the sdkms pods so that the proposed key is loaded from the database. This should be done using the CLI rolling restart mechanism described below. After the restart, the nodes are still using the old cluster master key.
```
cd /opt/fortanix/sdkms/bin
./dsm_backend_rolling_restart.sh
```
Get the new pod IPs for the sdkms pods after the restart using the following command:
```
sudo -E kubectl get pods -lapp=sdkms -owide
```

2.7 Activate the Cluster Master Key

Make a REST API call to any one sdkms pod to activate the new CMK:

curl -kv https://<node-ip >:4444/cluster/v1/key/activate -X POST

2.8 Restart the `sdkms` Pods Again

Restart all the sdkms pods using the following command so that the new cluster master key is used.
```
cd /opt/fortanix/sdkms/bin/
./dsm_backend_rolling_restart.sh
```
At this point, all new data protected by the cluster will use a key hierarchy rooted at the new CMK after each sdkms pod is rebooted.
Again, get the new pod IPs for the sdkms pods after the restart using the following command:
```
sudo -E kubectl get pods -lapp=sdkms -owide
```

2.9 Verify the New Key

Check for the new cluster key version using the following command:
```
curl -k https://<any-pod-IP>:4444/cluster/v1/key
```
It should be 2 if initially, it was 1.
Run the following command to verify the cluster_key_protection value for the new CMK. It must update to local after successful migration from external CDK to external DKS mechanism:
```
curl -ks https://<node-ip>:4444/admin/v1/cluster | jq ".cluster_key_protection"
```
Expected output:
```
"local"
```
Verify that the key_id has been fetched for the second time using the Activity Logs in the detailed view of the Deployment Key in the Fortanix DSM UI.
Figure 4: Deployment key fetched