Fortanix Data Security Manager Software Pre-Upgrade Checks - Manual

1.0 Introduction

This article describes the prechecks to be performed by the System Administrator before performing a DSM (DSM) software upgrade.

2.0 Prechecks

The run_precheck.sh script provides three options during execution, and the user can select the most suitable one by entering the correct input number.

1. Remote – It executes the /opt/fortanix/sdkms/bin/check-dsm-health.sh script remotely on all the cluster nodes and fetches the related files, formats them, and displays the status on the screen.

2. Local – It executes the /opt/fortanix/sdkms/bin/check-dsm-health.sh script locally on the same machine.

3. IPMI – It verifies the Intelligent Platform Management Interface (IPMI) connectivity.

2.1 Option 1 – Remote

Perform the following steps to execute the Fortanix DSM precheck script remotely on all the nodes:

1. Run the following command to go to the dsm_prechecks directory:

   sudo su
   cd /opt/fortanix/sdkms/bin/dsm_prechecks

2. Run the following command to create the config.txt file:

   kubectl get no --no-headers -owide | awk '{print $6 }' > config.txt

3. In the parameters.txt file, update the values for the following parameters:

   REMOTE_USER=""
   AUTH_TYPE="PASSWORD" or "PRIVATE_KEY"
   PRIVATE_KEY_FILE=""
   NO_PASSWORD="true" or "false"

   Where,

   • REMOTE_USER refers to the name of the user. For example, administrator.

   • AUTH_TYPE can be either PASSWORD or PRIVATE_KEY.

   • PRIVATE_KEY_FILE refers to the path of the private key. This is only applicable if the user selects the value for AUTH_TYPE parameter as PRIVATE_KEY.

   • NO_PASSWORD can be either true or false based on the sudo profile of the remote user.

4. Run the following command to execute the Fortanix DSM prechecks:

   ./run_prechecks.sh

5. Enter 1 to select "remote".
   The screen will prompt for password only if the user has selected the AUTH_TYPE=PASSWORD in Step 3.

   NOTE

   Create a node_password.txt file if you want to provide a password in the format IP|PASSWORD through the file.

6. The user can check the detailed DSM prechecks logs in the following directory created by the script:

   /tmp/health_checks/remote_logs

2.2 Option 2 - Local

Perform the following steps to execute the Fortanix DSM precheck script locally:

1. Run the following command to execute the Fortanix DSM prechecks:

   sudo ./run_prechecks.sh

2. Enter 2 to select "local".

3. The user can check the detailed Fortanix DSM precheck logs in the following directory created by the script:

   /tmp/health_checks/logs

2.3 Option 3 - IPMI

Perform the following steps to verify the Intelligent Platform Management Interface (IPMI) connectivity:

NOTE

Ensure that the IPMI is in the same network.

1. Run the following command to go to the dsm_prechecks directory:

   cd /opt/fortanix/sdkms/bin/dsm_prechecks

2. Run the following command to create the ipmi.txt file:

   vi ipmi.txt

3. Add IPMI address and username in the ipmi.txt file in the following format:

   10.10.10.10|admin
   11.11.11.11|admin

4. Run the following command to execute the Fortanix DSM prechecks:

   sudo ./run_prechecks.sh

5. Enter 3 to select "ipmi connectivity check".

2.4 Cluster Health Checks

After completing the prechecks, you can run the following cluster-level health check commands from any node within the cluster to validate its overall health:

These health checks help confirm that the cluster meets the necessary configuration, security, and readiness requirements before proceeding with the upgrade process.

The commands should be executed after the Fortanix DSM prechecks, and the results, along with the precheck outputs, must be shared with the Fortanix Support Team to verify the cluster’s health before the upgrade.

cat /etc/fortanix/sdkms_version/sdkms_version
kubectl get nodes,pods -owide -L datacenter
kubectl get pods -n kube-system -owide
kubectl get jobs
kubectl exec cassandra-0 -- nodetool status public;
kubectl get deployment -owide
kubectl get sts -owide
kubectl get ep -n swdist -owide
kubectl get cj -owide 
kubectl get po -o custom-columns=POD:.metadata.name,IMAGE:.spec.containers[*].image 
kubectl get po -o custom-columns=POD:.metadata.name,IMAGE:.spec.containers[*].image -n kube-system
sdkms-cluster get config --system | grep -v password

3.0 Handling Upgrade Issues Related to Attestation Value (Non-SGX Only)

If you are upgrading to a version lower than Fortanix DSM 4.31 and using a non-sgx build, check the value for the config-values parameter using the following command:

sdkms-cluster get config --system

If the attestation value is ias, then follow the workaround as outlined below. For more information, contact the Fortanix Support Team.

Perform the following steps:

1. Add the following parameter in the config.yaml file to update the attestation value:

   attestation: null

2. Run the following deploy command to apply the updated configurations:

   sdkms-cluster --config config.yaml --stage DEPLOY

   Monitor the deployment job and ensure all pods are running.

3. Run the following command to view the status of the pods:

   kubectl get pods –A -owide

4. Run the following command to verify the attestation value in the config-values file:

   kubectl get cm config-values -o yaml

   The following must be the output of the command:

   apiVersion: v1
   data:
     config-values: |-
       ---
       global:
         attestation: ~

4.0 Checks Handled by the Script

5.0 Fortanix DSM Prechecks Output Status

This section describes the following three type of precheck output status:

• OK: No action item and it is successful.

• WARN: Requires attention and the user needs to check the logs created on the node /tmp/health_checks/ and the user must share the log details with the Fortanix Support team.

• SKIPPED: Check was not executed due to some issue in the cluster.
  For example, Cassandra Replication Strategy check requires Cassandra to be healthy to fetch details from the Cassandra pod, but if the pod is not healthy, then that check will be SKIPPED.
  Hence, it is required to check the logs to understand the reason for skipping the check. The user must share the log details with the Fortanix support team.