Cluster Master Key Rotation

Introduction

This article describes the process of rotating the Cluster Master Key (CMK) in Fortanix DSM.

Cluster Master Key Rotation

Fortanix DSM uses a CMK to derive individual keys for functions such as database encryption.

The following section describes the steps to rotate the CMK using shell and kubectl on one of the Fortanix DSM nodes:

Propose a New Master Key

1. Get the pod IPs for the sdkms pods using the following command:

   sudo -E kubectl get pods -lapp=sdkms -owide

2. Propose a new CMK by making a REST API call to any one of the sdkms pods:

   curl -kv https://<pod-IP>:4444/cluster/v1/key/propose -X POST

   This should return 204 No Content.

Distribute the Proposed Cluster Master Key

Distribute the new CMK to the rest of the sdkms pods by making a REST API call to each of the other sdkms pods.

curl -kv https://<other-pod-IP>:4444/cluster/v1/key/fetch -X POST -d '{"target":"<pod-IP>:4444"}'

Where,

• pod-IP is the IP of the pod which received the propose API call.

• other-pod-IP is the IP of the rest of the other sdkms pods.

This should return 204 No Content.

Verify the New Cluster Master Key

For each sdkms pod, get the cluster master key status:

curl -k https://<pod-IP>:4444/cluster/v1/key

Each pod should return a fetched_proposal_hash JSON field and the field's value should be the same on each pod.

Restart the sdkms Pods

• Restart all the sdkmspods so that the proposed key is loaded from the database. This should be done using the CLI rolling restart mechanism described below. After the restart, the nodes are still using the old cluster master key.

  cd /opt/fortanix/sdkms/bin/
  ./dsm_backend_rolling_restart.sh

• Get the new pod IPs for the sdkms pods after the restart using the following command:

  sudo -E kubectl get pods -lapp=sdkms -owide

Activate the Cluster Master Key

Make a REST API call to any one sdkms pod:

curl -kv https://<any-pod-IP>:4444/cluster/v1/key/activate -X POST

This should return 204 No Content.

Restart the sdkms Pods Again

• Restart all the sdkmspods using the following command so that the new cluster master key is used.

  cd /opt/fortanix/sdkms/bin/ 
  ./dsm_backend_rolling_restart.sh

  At this point, all new data protected by the cluster will use a key hierarchy rooted at the new CMK after each sdkms pod is rebooted.

• Again, get the new pod IPs for the sdkms pods after the restart using the following command:

  sudo -E kubectl get pods -lapp=sdkms -owide
  

Verify the New Key

Check for the new cluster key version using the following command:

curl -k https://<any-pod-IP>:4444/cluster/v1/key

it should be 2 if initially, it was 1.