Fortanix DSM with Dell ObjectScale Using KMIP

1.0 Introduction

This document describes how to integrate Fortanix Data Security Manager (DSM) with Dell ObjectScale as an external key manager (EKM) using the Key Management Interoperability Protocol (KMIP).

Dell ObjectScale uses Fortanix DSM to manage the Control Key for server-side data-at-rest encryption (D@RE). Fortanix DSM creates and manages the top-level Control Key, while Dell ObjectScale communicates with DSM over KMIP to activate the key manager, rotate the Control Key, and reload the key after a service restart.

2.0 Why Use Fortanix DSM with Dell ObjectScale?

Fortanix DSM enables Dell ObjectScale to store the Control Key outside the storage platform in a centrally managed, FIPS 140-2 Level 3 validated key management system. By separating key management from data storage, organizations can strengthen their security posture and support regulatory and compliance requirements. Fortanix DSM also centralizes Control Key lifecycle management, allowing administrators to create, rotate, deactivate, and audit Control Keys from a single platform.

Only the Control Key is managed by Fortanix DSM. The encryption keys that protect object data are generated and stored within Dell ObjectScale. As a result, Dell ObjectScale communicates with Fortanix DSM only during key manager activation, Control Key rotation, and service restart, not during normal object read or write operations.

3.0 Architecture Workflow

The above architecture illustrates the integration between Dell ObjectScale and Fortanix DSM. Dell ObjectScale communicates with Fortanix DSM over the Key Management Interoperability Protocol (KMIP) to manage the Control Key used for server-side data-at-rest encryption (D@RE). While Fortanix DSM stores and manages the Control Key, the remaining encryption keys are generated and managed within Dell ObjectScale.

3.1 Dell ObjectScale Key Hierarchy

NOTE

The Virtual Control Key is derived from both the Control Key stored in Fortanix DSM and the RT Data Encryption Key stored locally in Dell ObjectScale. Because both keys are required to derive the Virtual Control Key, neither Fortanix DSM nor Dell ObjectScale alone can generate the complete encryption hierarchy.

3.2 Workflow

The following workflow describes the events when Dell ObjectScale communicates with Fortanix DSM during the Control Key lifecycle.

• Cluster activation: Dell ObjectScale creates the Control Key in Fortanix DSM over KMIP. As part of the activation process, Dell ObjectScale automatically initiates the initial Control Key rotation.

• Control Key rotation: Dell ObjectScale retrieves the latest generated Control Key from Fortanix DSM and rewraps the namespace keys by using the new Control Key. The previous Control Key generation is deactivated but retained, allowing existing encrypted data to remain accessible.

• Service restart: After a Dell ObjectScale service restart, Dell ObjectScale retrieves the Control Key from Fortanix DSM and re-derives the Virtual Control Key before resuming encryption and decryption operations.

During normal object read and write operations, Dell ObjectScale performs encryption and decryption locally. Fortanix DSM is contacted only for Control Key lifecycle operations and is not involved in normal object read or write requests.

NOTE

Dell ObjectScale 4.3.0 documentation refers to the top-level encryption key as the Control Key, while earlier Dell ECS documentation refers to the same key as the Master Key. This document uses the term Control Key throughout.

4.0 Prerequisites

Ensure the following:

• Fortanix DSM KMIP endpoint is reachable from the Dell ObjectScale nodes.

• Dell ObjectScale server-side encryption (D@RE) entitlement is applied to the deployment.

• Administrative access to both the Fortanix DSM console and Dell ObjectScale portal.

• AWS CLI and OpenSSL are installed on a client host or the verification procedures described in this guide.

5.0 Product Version Tested

The following product versions were tested:

• Fortanix DSM version 5.8 or later

• Dell ObjectScale version 4.3.0 or later

• AWS CLI version 1.23.2 (botocore 1.23.46, Python 2.7.5)

• OpenSSL version 1.0.2k-fips

• Client operating system RHEL or CentOS 7 (kernel 3.10.0-1160)

6.0 Port Requirements

Before you begin, verify that a Dell ObjectScale node can reach the Fortanix DSM KMIP endpoint. Run the following command from a Dell ObjectScale node:

timeout 5 bash -c "/5696" && echo "KMIP endpoint reachable" || echo "unreachable"

Output:

KMIP endpoint reachable

7.0 Configure Fortanix DSM

A Fortanix DSM service must be configured, and the URL must be accessible. To create a Fortanix DSM account and group, refer to the following sections:

7.1 Signing Up

7.2 Creating an Account

Access <Your_DSM_Service_URL> in a web browser and enter your credentials to log in to Fortanix DSM.

7.3 Creating a Group

Perform the following steps to create a group in the Fortanix DSM:

1. In the DSM left navigation panel, click the Groups menu item, and then click ADD GROUP to create a new group.

   

2. On the Adding new group page:

   1. Title: Enter a name for your group.

   2. Description (optional): Enter a short description of the group.

3. Click SAVE to create a new group.

7.4 Creating an Application

Perform the following steps to create an application (app) in the Fortanix DSM:

1. In the DSM left navigation panel, click the Apps menu item, and then click ADD APP to create a new app.

   

2. On the Adding new app page:

   1. App name: Enter the name for your application.

   2. ADD DESCRIPTION (optional): Enter a short description of the application.

   3. Authentication method: Select Certificate as the authentication method from the drop down menu. For more information on these authentication methods, refer to the User's Guide: Authentication.

   4. Assigning the new app to groups: Select the group created in Section 7.3: Creating a Groupfrom the list.

3. Click SAVE to add the new application.

7.5 Copying the App UUID

Perform the following steps to copy the app UUID from the Fortanix DSM:

1. Click the Apps menu item in the DSM left navigation panel and click the app created in Section 7.4: Creating an Application to go to the detailed view of the app.

2. From the top of the app's page, click the copy icon next to the app UUID to copy it. This value is used as the Common Name (CN) when generating the client certificate in Section 7.6: Generating the Client Certificate.

7.6 Generating the Client Certificate

Perform the following steps to generate the client certificate and package it into a PKCS#12 file for certificate-based authentication:

1. Run the following command to generate the private key and a Certificate Signing Request (CSR) with the application UUID as the Common Name (CN):

   openssl req -new -newkey rsa:2048 -nodes \
     -keyout client.key -out client.csr \
     -subj "/CN=<dsm-application-uuid>"

   Output:

   Generating a 2048 bit RSA private key
   ...........................+++
   .................+++
   writing new private key to 'client.key'
   -----

2. If the CSR is signed by a public or enterprise CA, use the signed certificate (client.crt) returned by the CA. Run the following command to generate the client certificate by self-signing the CSR:

   openssl x509 -req -in client.csr -signkey client.key -days 730 -out client.crt

   Output:

   Signature ok
   subject=/CN=<dsm-application-uuid>
   Getting Private key

3. Run the following command to package the client certificate and private key into a PKCS#12 file:

   openssl pkcs12 -export -inkey client.key -in client.crt \
     -out client.p12 -passout pass:<p12-password>

   Output:

   (completes silently and writes client.p12)

7.7 Updating the Authentication Method

Perform the following steps to change the authentication method:

1. Go to the detailed view of the app created in Section 7.4: Creating an Application and click Change authentication method and select the Certificate option to change the authentication method to Certificate.

2. Click SAVE.

3. On the Add certificate dialog box, click UPLOAD NEW CERTIFICATE to upload the certificate file or paste the content of the cert.pem certificate generated in Section 7.6: Generating the Client Certificate.

4. Select both check boxes to confirm your understanding of the action.

5. Click UPDATE to save the changes.

7.8 Configuring the KMIP Interface

Perform the following steps to configure the Fortanix DSM KMIP interface and establish the certificate trust required for Dell ObjectScale to communicate with Fortanix DSM:

1. This endpoint is required when configuring Dell ObjectScale.

2. Download the Certificate Authority (CA) certificate that signs the Fortanix DSM KMIP server certificate. Dell ObjectScale uses this CA certificate to validate the Fortanix DSM KMIP server certificate during every TLS handshake. Use either of the following methods:

   • Export the CA certificate from the Fortanix DSM Console:

     1. Open the KMIP endpoint or cluster certificate settings in the Fortanix DSM Console.

     2. Download the CA certificate or the complete certificate chain that signs the KMIP server certificate.

     3. Save the certificate as dsm-kmip-ca.pem.

     NOTE

     If the KMIP endpoint uses a publicly trusted certificate, download the complete issuer certificate chain, including any intermediate and root CA certificates.

   • Capture the certificate chain from the KMIP endpoint: Run the following command to retrieve the certificates presented by the Fortanix DSM KMIP endpoint on port 5696 and save the issuing CA certificate chain as dsm-kmip-ca.pem:

     openssl s_client -connect <dsm-kmip-endpoint>:5696 -showcerts </dev/null 2>/dev/null \
       | awk '/BEGIN CERTIFICATE/{c++} c>1' > dsm-kmip-ca.pem
     echo "captured $(grep -c 'BEGIN CERTIFICATE' dsm-kmip-ca.pem) CA certificate(s)"

     Output:

     captured 2 CA certificate(s)

3. Run the following command to verify that the downloaded CA certificate successfully validates the Fortanix DSM KMIP server certificate before configuring Dell ObjectScale:

   openssl s_client -connect <dsm-kmip-endpoint>:5696 -CAfile dsm-kmip-ca.pem </dev/null 2>/dev/null \
     | grep -E 'Verify return code|subject='

   Output:

   subject=/CN=<dsm-application-uuid>
   Verify return code: 0 (ok)

   Use the dsm-kmip-ca.pem file as the certificate_authority value when adding the DSM KMIP server in Section 8.3: Adding the DSM KMIP Server to the Cluster.

4. Ensure that the application is granted only the following KMIP operations required by the Dell ObjectScale integration:

   • Create

   • Activate

   • Get

   • Get Attributes

   • Modify Attribute (for deactivation)

   Ensure that no broader permissions are granted. This follows the principle of least privilege by limiting the KMIP application to only the operations required for the EKM workflow.

   NOTE

   Dell ObjectScale creates the Control Key automatically during cluster activation. You do not need to create the key manually in Fortanix DSM. This procedure prepares the application, the KMIP endpoint, and mutual TLS trust required for Dell ObjectScale to communicate securely with Fortanix DSM.

8.0 Configure Dell ObjectScale

This section describes the steps to configure Dell ObjectScale to use the Fortanix DSM KMIP endpoint as an EKM of the CipherTrust type, map the EKM cluster to the Virtual Data Center (VDC), and activate the cluster.

This procedure uses the Dell ObjectScale management API on port 4443. You can also perform the same configuration by using the Dell ObjectScale portal.

NOTE

Dell ObjectScale supports the following external key manager types:

• Thales CipherTrust Manager

• IBM Security Key Lifecycle Manager (SKLM)

• Gemalto/SafeNet KeySecure.

Although Fortanix DSM is not listed as a supported external key manager type, it is compatible with the Dell ObjectScale CipherTrust (KMIP v1.4) profile. Therfore, configure the Fortanix DSM KMIP endpoint as a CIPHERTRUST cluster. Dell ObjectScale uses client certificate authentication for the CipherTrust profile, which matches the certificate-based authentication configured for the Fortanix DSM application in in Section 7.4: Creating an Application.

8.1 Obtaining a Management Session Token

Run the following command to obtain a Dell ObjectScale management session token and set the node and cluster variables that is used throughout this section:

export OS_NODE=<objectscale-node>
TOKEN=$(curl -sk -u <username>:<password> -D - https://${OS_NODE}:4443/login \
  | awk -F': ' '/X-SDS-AUTH-TOKEN/{gsub("\r","",$2); print $2}')
echo "token acquired: ${TOKEN:+yes}"

Output:

token acquired: yes

8.2 Creating the External Key Manager Cluster

Run the following command to create an external key manager cluster of type CIPHERTRUST:

curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -H "Accept: application/json" \
  -H "Content-Type: application/json" -X POST \
  https://${OS_NODE}:4443/ekm/cluster \
  -d '{"name":"objectscale-dsm-ekm","cluster_type":"CIPHERTRUST"}'

Output:

{
  "name": "objectscale-dsm-ekm",
  "id": "<ekm-cluster-id>",
  "cluster_type": "CIPHERTRUST",
  "status": "CREATED"
}

Note the returned cluster ID and export it:

export CLUSTER_ID=<ekm-cluster-id>

8.3 Adding the DSM KMIP Server to the Cluster

Run the following command to add the Fortanix DSM KMIP server to the Dell ObjectScale EKM cluster:

CA_PEM=$(cat dsm-kmip-ca.pem)
P12_B64=$(base64 -w0 client.p12) 
curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -H "Accept: application/json" \
  -H "Content-Type: application/json" -X POST \
  https://${OS_NODE}:4443/ekm/server \
  -d "$(jq -n \
    --arg cluster_id "${CLUSTER_ID}" \
    --arg server_hostname "dsm-kmip.example.com" \
    --arg fqdn_ip "<dsm-ip>" \
    --argjson port 5696 \
    --arg certificate_authority "${CA_PEM}" \
    --arg identity_store "${P12_B64}" \
    --arg identity_store_password "<p12-password>" \
    --arg username "<dsm-application-uuid>" \
    '{cluster_id:$cluster_id, server_hostname:$server_hostname, fqdn_ip:$fqdn_ip,
      port:$port, certificate_authority:$certificate_authority,
      certificate_revocation_list:"", identity_store:$identity_store,
      identity_store_password:$identity_store_password, username:$username}')"

Where,

• Username: The Fortanix DSM application UUID as copied in Section 7.5: Copying the App UUID.

• certificate_authority: The Fortanix DSM CA certificate (dsm-kmip-ca.pem) as downloaded in Section 7.8: Configuring the KMIP Interface.

• identity_store : The Base64-encoded PKCS#12 (client.p12) as generated in Section 7.6: Generating the Client Certificate.

Output:

{
  "id": "urn:storageos:EKMServer:<ekm-server-1-uuid>",
  "cluster_id": "<ekm-cluster-id>",
  "server_hostname": "dsm-kmip.example.com",
  "fqdn_ip": "<dsm-ip>",
  "port": 5696,
  "username": "<dsm-application-uuid>"
}

Repeat this command to add the second Fortanix DSM KMIP server entry that reaches the same Fortanix DSM key state. Record the server URNs returned by both operations

The following table maps the Dell ObjectScale API parameters to the corresponding server properties in the Dell ObjectScale portal:

8.4 Mapping the EKM Servers to a Virtual Data Center

NOTE

• Dell ObjectScale requires at least two EKM server entries to be mapped to a VDC. The first mapped server acts as the primary server for Control Key creation and retrieval, while the remaining server(s) act as backup servers for key retrieval if the primary server becomes unavailable.

• During cluster activation, Dell ObjectScale verifies that all mapped EKM servers are reachable and that the same Control Key can be retrieved from each server. Therefore, all mapped servers must reference the same Fortanix DSM key state. If your Fortanix DSM deployment exposes a single KMIP endpoint, add the same endpoint as two server entries to satisfy the minimum two-server requirement while maintaining a single shared key state.

• Do not map endpoints from different Fortanix DSM clusters. Instead, map two endpoints of the same Fortanix DSM high-availability (HA) deployment that shares a common key store. If the mapped EKM server entries do not reference the same Fortanix DSM key state, Dell ObjectScale cannot retrieve the Control Key from every mapped server, and cluster activation fails.

Run the following command to map the Fortanix KMIP servers to the VDC:

export VDC_ID=<vdc-id>
curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -H "Accept: application/json" \
  -H "Content-Type: application/json" -X PUT \
  https://${OS_NODE}:4443/ekm/cluster/${CLUSTER_ID} \
  -d "$(jq -n \
    --arg name "objectscale-dsm-ekm" \
    --arg vdc_id "${VDC_ID}" \
    --arg s1 "<ekm-server-1-urn>" \
    --arg s2 "<ekm-server-2-urn>" \
    '{name:$name, ekm_mapping_set:[{vdc_id:$vdc_id, ekm_servers_list:[$s1,$s2]}]}')"

Output:

{
  "name": "objectscale-dsm-ekm",
  "id": "<ekm-cluster-id>",
  "cluster_type": "CIPHERTRUST",
  "ekm_mapping_set": [
    {
      "vdc_id": "<vdc-id>",
      "ekm_servers_list": ["<ekm-server-1-urn>", "<ekm-server-2-urn>"]
    }
  ]
}

If you do not know the VDC ID, retrieve it by running the following command:

curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -H "Accept: application/json" https://${OS_NODE}:4443/object/vdcs/vdc/local

8.5 Activating the EKM Cluster

Run the following command to activate the Dell ObjectScale EKM cluster:

curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -H "Accept: application/json" \
  -X POST https://${OS_NODE}:4443/ekm/cluster/${CLUSTER_ID}/activate

Output:

{
  "id": "<ekm-cluster-id>",
  "status": "ACTIVATION_INITIATED",
  "activation_message": "EKM cluster activation initiated"
}

NOTE

Cluster activation is an asynchronous operation. During activation, Dell ObjectScale creates the Control Key in Fortanix DSM, performs the initial key rotation, and verifies the Control Key across all mapped EKM servers. The cluster status changes to ACTIVATED after these operations complete successfully. To verify the activation status, follow the procedure in Section 8.6: Verifying the EKM Cluster Activation until the status is ACTIVATED.

8.6 Verifying the EKM Cluster Activation

Run the following command to verify that the that the Dell ObjectScale EKM cluster has reached the ACTIVATED state:

curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -H "Accept: application/json" \
  https://${OS_NODE}:4443/ekm/cluster/${CLUSTER_ID}

Output:

{
  "name": "objectscale-dsm-ekm",
  "cluster_type": "CIPHERTRUST",
  "status": "ACTIVATED",
  "activation_message": "EKMCluster <ekm-cluster-id> has been activated",
  "ekm_mapping_set": [
    {
      "vdc_id": "<vdc-id>",
      "ekm_servers_list": ["<ekm-server-1-urn>", "<ekm-server-2-urn>"]
    }
  ]
}

A successful activation verifies that Dell ObjectScale has established a successful KMIP over TLS connection to the Fortanix DSM KMIP endpoint and that all mapped EKM servers are healthy. It also confirms that the client credentials, certificate trust, and two-server mapping are configured correctly.

9.0 Verify the Integration

Perform the following procedures to verify the Dell ObjectScale and Fortanix DSM after configuration is complete.

These procedures verify, key management operations, encryption behavior, Control Key rotation, certificate trust, and recovery scenarios to ensure that Dell ObjectScale communicates successfully with the Fortanix DSM KMIP endpoint.

9.1 Control Key Creation and Initial Rotation

As part of cluster activation, Dell ObjectScale creates the Control Key in Fortanix DSM over KMIP and automatically initiates the initial key rotation.

1. In the Fortanix DSM UI, navigate to the Security Objects table. Verify that Dell ObjectScale successfully created the AES-256 Control Key in Fortanix DSM during cluster activation and initiated the initial Control Key rotation. This security object is the Dell ObjectScale Control Key. Verify that its state is Active.

   

2. Run the following command to verify that the rotation task was created in Dell ObjectScale:

   curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -H "Accept: application/json" \
     https://${OS_NODE}:4443/rotationtask

   Output:

   {
     "rotation_task_set": [
       { "id": "<rotation-task-id>" }
     ]
   }

3. In the Fortanix DSM audit log for the application, verify that the Create and Activate operations for the Control Key were recorded at the time of cluster activation. These operations confirm that Dell ObjectScale successfully created and activated the Control Key through the KMIP interface.

   

9.2 Encrypted Object Operations

Verify that objects stored in an existing encryption-enabled bucket are encrypted by using the Dell ObjectScale D@RE hierarchy rooted in the Fortanix DSM Control Key. Before performing this procedure, ensure that an encryption-enabled bucket is available.

1. Run the following command to set the AWS access credentials:

   export AWS_ACCESS_KEY_ID=<access-key>
   export AWS_SECRET_ACCESS_KEY=<secret-key>

2. Run the following command to upload an object to an encryption-enabled bucket:

   aws s3api put-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket encrypted-bucket --key testobject --body ./testfile

   Output:

   {
     "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
     "ServerSideEncryption": "AES256"
   }

   Verify that the ServerSideEncryption field is set to AES256. This confirms that server-side Encryption is enabled for the object.

3. Run the following command to download the object and compare it with the original file:

   aws s3api get-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket encrypted-bucket --key testobject ./testfile.out
   cmp ./testfile ./testfile.out && echo "byte-identical"

   Output:

   byte-identical

Verify that the downloaded file is identical to the original file. This confirms that the object was encrypted and decrypted successfully by using the Dell ObjectScale D@RE hierarchy rooted in the Fortanix DSM Control Key.

9.3 Key Rotation

Verify that Dell ObjectScale successfully rotates the Control Key and continue to decrypt existing objects after the rotation.

1. Run the following command to trigger a Control Key rotation:

   curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -H "Accept: application/json" \
     -X POST https://${OS_NODE}:4443/rotationtask

   Output:

   {
     "id": "<rotation-task-id>",
     "status": "STARTED",
     "user_id": "<user>"
   }

2. After the rotation completes, run the following command to download the same object and compare it with the original file:

   aws s3api get-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket encrypted-bucket --key testobject ./after-rotation.out
   cmp ./testfile ./after-rotation.out && echo "byte-identical after rotation"

   Output:

   byte-identical after rotation

   Verify that the downloaded file is identical to the original file. This confirms that existing objects remain accessible after the Control Key rotation.

3. Verify that the Fortanix DSM audit log shows that a new version of Control Key was created and the previous version was deactivated. Verify that the previous version has a Deactivation Date. The existing data remains accessible because the previous key version is deactivated, not destroyed.

9.4 Certificate Configuration

Verify that Dell ObjectScale successfully validates the Fortanix DSM KMIP endpoint server certificate by using the configured CA and connections presenting server certificates signed by untrusted or expired CA are rejected.

1. Run the following command to verify that a connection using an incorrect CA certificate is rejected:

   openssl s_client -connect dsm-kmip.example.com:5696 -CAfile wrong-ca.pem </dev/null 2>/dev/null \
     | openssl verify 2>&1 | head -1

   Output:

   verify error:num=20:unable to get local issuer certificate

   Verify that certificate validation fails because the server certificate cannot be verified against the supplied CA certificate.

2. Run the following command to verify that the Fortanix DSM KMIP endpoint is trusted by using the correct CA certificate:

   echo | openssl s_client -connect dsm-kmip.example.com:5696 -CAfile dsm-kmip-ca.pem 2>/dev/null \
     | grep -E "Verify return code"

   Output:

   Verify return code: 0 (ok)

   Verify that the KMIP server certificate is validated successfully against the trusted CA certificate.

3. To renew an expiring certificate, replace the CA or server certificate in Fortanix DSM, update the trusted CA certificate on the Dell ObjectScale cluster, and verify that the cluster returns to the HEALTHY state.

9.5 Key Availability During Fortanix DSM Unavailability (Optional)

Verify that Dell ObjectScale continues to serve encrypted objects during a temporary loss of connectivity to Fortanix DSM because the Control Key remains cached for the lifetime of the Dell ObjectScale service.

1. Run the following command to block outbound traffic from the Dell ObjectScale node to the Fortanix DSM KMIP endpoint:

   sudo iptables -I OUTPUT -p tcp -d <dsm-ip> --dport 5696 -j DROP

2. Run the following command to verify that the KMIP endpoint is unreachable:

   timeout 5 bash -c "</dev/tcp/<dsm-ip>/5696" && echo reachable || echo unreachable

   Output:

   Unreachable

3. Run the following command to read an existing encrypted object:

   aws s3api get-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket encrypted-bucket --key testobject ./cached-read.out && echo "served from cache"

   Output:

   served from cache

   Verify that the object is retrieved successfully. This confirms that Dell ObjectScale continues to serve encrypted data by using the cached Control Key while Fortanix DSM is temporarily unavailable.

4. Run the following command to restore connectivity to the Fortanix DSM KMIP endpoint:

   sudo iptables -D OUTPUT -p tcp -d <dsm-ip> --dport 5696 -j DROP

   Output:

   served from cache

NOTE

If the Dell ObjectScale service is restarted while Fortanix DSM is unreachable, the cached Control Key is cleared and encrypted object reads fail until connectivity to Fortanix DSM is restored. After connectivity is restored, Dell ObjectScale retrieves and re-caches the Control Key. Ensure that the Fortanix DSM KMIP endpoint is reachable before restarting Dell ObjectScale services.

9.6 Key Rotation with Concurrent I/O

Verify that Dell ObjectScale continues to write and retrieve objects successfully while a Control Key rotation is in progress.

1. Run the following command to create a set of test files, upload them to an encryption-enabled bucket in the background, and trigger a Control Key rotation while the uploads are in progress:

   for i in $(seq 1 20); do printf 'load %s' "$i" > /tmp/load-$i.txt; done
   for i in $(seq 1 20); do
     aws s3api put-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
       --bucket encrypted-bucket --key load-$i.txt --body /tmp/load-$i.txt >/dev/null &
   done
   curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -X POST https://${OS_NODE}:4443/rotationtask >/dev/null
   wait

2. Run the following command to download each object and compare it with the original file:

   for i in $(seq 1 20); do
     aws s3api get-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
       --bucket encrypted-bucket --key load-$i.txt /tmp/load-$i.out >/dev/null \
       && cmp -s /tmp/load-$i.txt /tmp/load-$i.out && echo "load-$i ok"
   done

   Output:

   load-1 ok
   load-2 ok
   ...
   load-20 ok

Verify that every object is downloaded successfully and matches the original file. This confirms that concurrent object writes remain consistent during the Control Key rotation.

9.7 Multipart Upload Across a Rotation

Verify that a multipart upload completes successfully when a Control Key rotation occurs between part uploads and that the assembled object remains intact.

1. Run the following command to create the multipart upload, upload the first part, trigger a Control Key rotation, upload the second part, and complete the multipart upload:

   dd if=/dev/urandom of=/tmp/part1 bs=1M count=5 2>/dev/null
   dd if=/dev/urandom of=/tmp/part2 bs=1k count=17 2>/dev/null
   cat /tmp/part1 /tmp/part2 > /tmp/mpu.src
   UPLOAD_ID=$(aws s3api create-multipart-upload --endpoint-url https://${OS_NODE}:9021 \
     --no-verify-ssl --bucket encrypted-bucket --key mpu.bin --query UploadId --output text)
   E1=$(aws s3api upload-part --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket encrypted-bucket --key mpu.bin --upload-id "$UPLOAD_ID" \
     --part-number 1 --body /tmp/part1 --query ETag --output text)
   curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -X POST \
     https://${OS_NODE}:4443/rotationtask >/dev/null
   E2=$(aws s3api upload-part --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket encrypted-bucket --key mpu.bin --upload-id "$UPLOAD_ID" \
     --part-number 2 --body /tmp/part2 --query ETag --output text)
   aws s3api complete-multipart-upload --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket encrypted-bucket --key mpu.bin --upload-id "$UPLOAD_ID" \
     --multipart-upload "{\"Parts\":[{\"PartNumber\":1,\"ETag\":$E1},{\"PartNumber\":2,\"ETag\":$E2}]}" >/dev/null

2. Run the following command to download the completed object and compare it with the original file:

   aws s3api get-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket encrypted-bucket --key mpu.bin /tmp/mpu.out >/dev/null
   cmp /tmp/mpu.src /tmp/mpu.out && echo "multipart object byte-identical across rotation"

   Output:

   multipart object byte-identical across rotation

Verify that the downloaded object is identical to the original file. This confirms that the multipart upload completed successfully and that the Control Key rotation did not affect object integrity.

9.8 Versioned Objects Across a Rotation

Verify that all versions of an object remain accessible after a Control Key rotation.

1. Run the following command to enable bucket versioning, upload two versions of the same object, and trigger a Control Key rotation:

   aws s3api put-bucket-versioning --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket encrypted-bucket --versioning-configuration Status=Enabled
   printf 'version one' > /tmp/v1.txt
   printf 'version two' > /tmp/v2.txt
   V1=$(aws s3api put-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket encrypted-bucket --key ver.txt --body /tmp/v1.txt --query VersionId --output text)
   V2=$(aws s3api put-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket encrypted-bucket --key ver.txt --body /tmp/v2.txt --query VersionId --output text)
   curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -X POST \
     https://${OS_NODE}:4443/rotationtask >/dev/null

2. Run the following command to retrieve both object versions and compare them with the original files:

   aws s3api get-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket encrypted-bucket --key ver.txt --version-id "$V1" /tmp/g1.txt >/dev/null
   aws s3api get-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket encrypted-bucket --key ver.txt --version-id "$V2" /tmp/g2.txt >/dev/null
   cmp /tmp/v1.txt /tmp/g1.txt && \
   cmp /tmp/v2.txt /tmp/g2.txt && \
   echo "both versions intact across rotation"

   Output:

   both versions intact across rotation

Verify that both object versions match their original files. This confirms that the Control Key rotation does not affect access to existing object versions.

9.9 File System Bucket Across a Rotation

Verify that object upload and retrieval on an encrypted file system-enabled bucket continue to function correctly during a Control Key rotation.

1. Run the following command to create an encrypted file system-enabled bucket by using the Dell ObjectScale management API:

   curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -H "Accept: application/json" \
     -H "Content-Type: application/json" -H "x-emc-namespace: <namespace>" \
     -X POST https://${OS_NODE}:4443/object/bucket \
     -d '{"name":"fs-bucket","filesystem_enabled":true,"is_encryption_enabled":true}'

   Output:

   {
     "name": "fs-bucket",
     "id": "<namespace>.fs-bucket",
     "filesystem_enabled": true
   }

2. Run the following command to upload an object to the bucket, trigger a Control Key rotation, retrieve the object, and compare it with the original file:

   printf 'file-system bucket payload' > /tmp/fs.txt
   aws s3api put-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket fs-bucket --key fsobject --body /tmp/fs.txt
   curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -X POST \
     https://${OS_NODE}:4443/rotationtask >/dev/null
   aws s3api get-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket fs-bucket --key fsobject /tmp/fs.out >/dev/null
   cmp /tmp/fs.txt /tmp/fs.out && \
   echo "file-system bucket object byte-identical across rotation"

   Output:

   {
     "ETag": "\"6f9b3a...\"",
     "ServerSideEncryption": "AES256"
   }
   file-system bucket object byte-identical across rotation

Verify that the uploaded object is encrypted (ServerSideEncryption: "AES256") and that the downloaded object matches the original file. This confirms that encrypted file system-enabled buckets continue to function correctly during a Control Key rotation.

9.10 Cache Eviction After Service Restart

NOTE

This verification restarts a Dell ObjectScale service and temporarily interrupts object access. Perform this verification only during a maintenance window. Do not run it on a production cluster that is serving client traffic.

Verify that restarting the Dell ObjectScale service clears the cached Control Key, causing encrypted object reads to fail while Fortanix DSM is unreachable, and that object access is restored after connectivity to Fortanix DSM is re-established.

1. Run the following command to block outbound traffic from the Dell ObjectScale node to the Fortanix DSM KMIP endpoint and verify that the endpoint is unreachable:

   sudo iptables -I OUTPUT -p tcp -d <dsm-ip> --dport 5696 -j DROP
   timeout 5 bash -c "</dev/tcp/<dsm-ip>/5696" || echo "EKM unreachable"

2. Restart the Dell ObjectScale object service by using your platform's supported restart procedure. Then run the following command to retrieve an encrypted object:

   aws s3api get-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl 
     --bucket encrypted-bucket --key testobject /tmp/evict.out \
     || echo "read failed: key cache cleared and EKM unreachable"

3. Run the following command to restore connectivity to the Fortanix DSM KMIP endpoint and retrieve the object again:

   sudo iptables -D OUTPUT -p tcp -d <dsm-ip> --dport 5696 -j DROP
   aws s3api get-object --endpoint-url https://${OS_NODE}:9021 --no-verify-ssl \
     --bucket encrypted-bucket --key testobject /tmp/recache.out \
     && echo "read succeeded after re-cache"

   Output:

   EKM unreachable
   read failed: key cache cleared and EKM unreachable
   read succeeded after re-cache

Verify that the initial read fails because the cached Control Key is no longer available while Fortanix DSM and Fortanix DSM is unreachable. This confirms that encrypted object reads depend on the cached Control Key when Fortanix DSM is unavailable. After connectivity is restored, verify that Dell ObjectScale retrieves and re-caches the Control Key, allowing encrypted object reads to succeed.

9.11 EKM Configuration

Verify the current external key manager (EKM) cluster configuration, activation status, and VDC mapping.

Run the following command to retrieve the EKM cluster configuration:

curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -H "Accept: application/json" \
  https://${OS_NODE}:4443/ekm/cluster/${CLUSTER_ID}

Output:

{
  "name": "objectscale-dsm-ekm",
  "cluster_type": "CIPHERTRUST",
  "status": "ACTIVATED",
  "ekm_mapping_set": [
    {
      "vdc_id": "<vdc-id>",
      "ekm_servers_list": [
        "<ekm-server-1-urn>",
        "<ekm-server-2-urn>"
      ]
    }
  ]
}

Verify that the EKM cluster type is CIPHERTRUST, the cluster status is ACTIVATED, and the expected VDC and EKM server mappings are configured correctly.

10.0 Decommission the Integration

This section describes how to remove Fortanix DSM integration from Dell ObjectScale by deactivating the EKM cluster, removing the VDC mapping, and deleting the EKM cluster.

NOTE

Perform this procedure only after verifying that no Dell ObjectScale data depends on the Fortanix DSM Control Key. Removing the EKM configuration while encrypted data still depends on the Control Key makes that data inaccessible.

1. Run the following command to deactivate the EKM cluster:

   curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -H "Accept: application/json" \
     -X POST https://${OS_NODE}:4443/ekm/cluster/${CLUSTER_ID}/deactivate

   Output:

   {
     "id": "<ekm-cluster-id>",
     "status": "DEACTIVATED"
   }

   Verify that the cluster status changes to DEACTIVATED.

2. Run the following command to remove the EKM server mapping from the VDC:

   curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -H "Accept: application/json" \
     -H "Content-Type: application/json" -X PUT \
     https://${OS_NODE}:4443/ekm/cluster/${CLUSTER_ID} \
     -d "$(jq -n --arg vdc_id "${VDC_ID}" \
       '{ekm_mapping_set:[{vdc_id:$vdc_id, ekm_servers_list:[]}]}')"

   Output:

   {
     "name": "objectscale-dsm-ekm",
     "id": "<ekm-cluster-id>",
     "ekm_mapping_set": [
       {
         "vdc_id": "<vdc-id>",
         "ekm_servers_list": []
       }
     ]
   }

   Verify that the ekm_servers_list array is empty, indicating that all EKM server mappings have been removed from the VDC.

3. Run the following command to delete the EKM cluster:

   curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -H "Accept: application/json" \
   
     -X DELETE https://${OS_NODE}:4443/ekm/cluster/${CLUSTER_ID}

   Output:

   HTTP 200

   Verify that the EKM cluster and its associated EKM server entries have been removed successfully.

NOTE

You can retain the Control Key in Fortanix DSM for audit or recovery purposes, or delete it according to your organization's key retention policy after confirming that no Dell ObjectScale data depends on it.

11.0 Troubleshooting

The following table describes common issues that you may encounter when configuring the Dell ObjectScale and Fortanix DSM integration, along with possible causes and recommended resolutions.

curl -sk -H "X-SDS-AUTH-TOKEN:${TOKEN}" -H "Accept: application/json" https://${OS_NODE}:4443/rotationtask

{"rotation_task_set":[{"id":"<rotation-task-id>"}]}