Software Post-Upgrade Checks

1.0 Introduction

This article describes the checks to be performed by the System Administrator after upgrading the Fortanix-Data-Security-Manager (DSM) software.

2.0 Post Upgrade Checks

The run_health_checks.sh script provides three options after execution, and the user can select the most suitable one by entering the correct input number.

Remote – It executes the /opt/fortanix/sdkms/bin/check-dsm-health.sh script remotely on all the cluster nodes and fetches the related files, formats them, and displays the status on the screen.
Local – It executes the /opt/fortanix/sdkms/bin/check-dsm-health.sh script locally on the same machine.
IPMI – It verifies the Intelligent Platform Management Interface (IPMI) connectivity.

2.1 Option 1 – Remote

Perform the following steps to execute the Fortanix DSM post check script remotely on all the nodes:

Run the following command to go to the dsm_healthchecks directory:
```
sudo su
cd /opt/fortanix/sdkms/bin/dsm_healthchecks
```

Run the following command to create the config.txt file:

kubectl get no --no-headers -owide | awk '{print $6 }' > config.txt

In the parameters.txt file, update the values for the following parameters:
```
REMOTE_USER=""
AUTH_TYPE="PASSWORD" or "PRIVATE_KEY"
PRIVATE_KEY_FILE=""
NO_PASSWORD="true" or "false"
```
Where,
- REMOTE_USER refers to the name of the user. For example, administrator.
- AUTH_TYPE can be either PASSWORD or PRIVATE_KEY.
- PRIVATE_KEY_FILE refers to the path of the private key. This is only applicable if the user selects the value for AUTH_TYPE parameter as PRIVATE_KEY.
- NO_PASSWORD can be either true or false based on the sudo profile of the remote user.
Run the following command to execute the Fortanix DSM post upgrade checks:
```
./run_health_checks.sh
```
Enter 1 to select "remote".
The screen will prompt for password only if the user has selected the AUTH_TYPE=PASSWORD in Step 3.
NOTE
Create a node_password.txt file if you want to provide a password in the format IP|PASSWORD through the file.
The user can check the detailed DSM prechecks logs in the following directory created by the script:
```
/tmp/health_checks/remote_logs
```

2.2 Option 2 - Local

Perform the following steps to execute the Fortanix DSM post upgrade checks script locally:

Run the following command to execute the Fortanix DSM post upgrade checks:
```
sudo ./run_health_checks.sh
```
Enter 2 to select "local".
The user can check the detailed Fortanix DSM precheck logs in the following directory created by the script:
```
/tmp/health_checks/logs
```

2.3 Option 3 - IPMI

Perform the following steps to verify the Intelligent Platform Management Interface (IPMI) connectivity:

NOTE
Ensure that the IPMI is in the same network.

Run the following command to go to the dsm_healthchecks directory:
```
cd /opt/fortanix/sdkms/bin/dsm_healthchecks
```
Run the following command to create the ipmi.txt file:
```
vi ipmi.txt
```
Add IPMI address and username in the ipmi.txt file in the following format:
```
10.10.10.10|admin
11.11.11.11|admin
```
Run the following command to execute the Fortanix DSM post upgrade checks:
```
sudo ./run_health_checks.sh
```
Enter 3 to select "ipmi connectivity check".

2.4 Cluster Health Checks

After completing the post upgrade checks, you can run the following cluster-level health check commands from any node within the cluster to validate its overall health:

These health checks help confirm that the cluster meets the necessary configuration, security, and readiness requirements after the upgrade process.

The commands should be executed after the Fortanix DSM post checks, and the results, along with the post checks outputs, must be shared with the Fortanix Support Team to verify the cluster’s health after the upgrade.

cat /etc/fortanix/sdkms_version/sdkms_version
kubectl get nodes,pods -owide -L datacenter
kubectl get pods -n kube-system -owide
kubectl get jobs
kubectl exec cassandra-0 -- nodetool status public;
kubectl get deployment -owide
kubectl get sts -owide
kubectl get ep -n swdist -owide
kubectl get cj -owide 
kubectl get po -o custom-columns=POD:.metadata.name,IMAGE:.spec.containers[*].image 
kubectl get po -o custom-columns=POD:.metadata.name,IMAGE:.spec.containers[*].image -n kube-system
sdkms-cluster get config --system | grep -v password

2.5 Certificate Manager Verification

After the Fortanix DSM software is upgraded on your machine, perform the following steps to check if cert-manager is upgraded successfully:

Run the following command to check for all the resources in the cert-manager namespace:

kubectl get all -n cert-manager

The following is the sample output:

NAME                                                       READY   STATUS    RESTARTS      AGE
pod/cert-manager-csi-driver-hc7gr                          3/3     Running   4 (59m ago)   101m
pod/certmanager-cert-manager-6c6bdd85d9-kzh7b              1/1     Running   0             63m
pod/certmanager-cert-manager-cainjector-7b7cbc6988-6hp7d   1/1     Running   0             63m
pod/certmanager-cert-manager-webhook-555cbb78cd-t6w2r      1/1     Running   0             63m

NAME                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/certmanager-cert-manager           ClusterIP   10.245.241.3    <none>        9402/TCP   101m
service/certmanager-cert-manager-webhook   ClusterIP   10.245.190.11   <none>        443/TCP    101m

NAME                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/cert-manager-csi-driver   1         1         1       1            1           <none>          101m

NAME                                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/certmanager-cert-manager              1/1     1            1           101m
deployment.apps/certmanager-cert-manager-cainjector   1/1     1            1           101m
deployment.apps/certmanager-cert-manager-webhook      1/1     1            1           101m

NAME                                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/certmanager-cert-manager-6c6bdd85d9              1         1         1       101m
replicaset.apps/certmanager-cert-manager-cainjector-7b7cbc6988   1         1         1       101m
replicaset.apps/certmanager-cert-manager-webhook-555cbb78cd      1         1         1       101m

Run the following command to view the helm chart status of cert-manager and check the version number:

helm list -A

The following is the sample output:

NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                          APP VERSION
certmanager     cert-manager    10              2023-05-30 14:27:59.266416986 +0000 UTC deployed        cert-manager-v1.11.2           v1.11.2
csiplugin       cert-manager    10              2023-05-30 14:28:01.027483969 +0000 UTC deployed        cert-manager-csi-driver-v0.5.0 v0.5.0

NOTE
Ensure that the version number of cert-manager is 1.11.2.

Run the following command to view the list of cert-manager pods:

kubectl get pods -n cert-manager

The following is the sample output:

NAME                                                   READY   STATUS    RESTARTS      AGE
cert-manager-csi-driver-hc7gr                          3/3     Running   4 (53m ago)   95m
certmanager-cert-manager-6c6bdd85d9-kzh7b              1/1     Running   0             58m
certmanager-cert-manager-cainjector-7b7cbc6988-6hp7d   1/1     Running   0             58m
certmanager-cert-manager-webhook-555cbb78cd-t6w2r      1/1     Running   0             58m
root@ip-172-31-0-42:/home/administrator#

Run the following command to check the status of the Cassandra pods:
```
kubectl get pods | grep cassandra
```
The following is the sample output:
```
cassandra-0 1/1 Running 0 57m
```
Run the following command to check the value of DEPLOYMENT_STATUS variable in all the Cassandra pods:
```
kubectl exec -it cassandra-0 -- env | grep DEPLOYMENT_STAGE
DEPLOYMENT_STAGE=CERT_MANAGER_ONLY
```
NOTE
Ensure that the value is CERT_MANGER_ONLY.

Run the following command to check if the configmap is created or not with the name cassandra-cert-manager-migration-state:

kubectl get cm cassandra-cert-manager-migration-state
NAME                                     DATA   AGE
cassandra-cert-manager-migration-state   1      99m

kubectl get cm cassandra-cert-manager-migration-state -ojsonpath='{.data}'
{"DEPLOYMENT_STAGE":"CERT_MANAGER_ONLY"}

Run the following command to check the secret resource with the name cassandra-mtls-ca:
```
kubectl get secrets cassandra-mtls-ca
```
The following is the sample output:
```
NAME TYPE DATA AGE
cassandra-mtls-ca kubernetes.io/tls 3 100m
```

Save the following code snippet to a required file:

#!/bin/bash
cass_num=$(kubectl get statefulsets cassandra -ojsonpath='{.status.replicas}')
cass_num=$((cass_num-1))
for index in $(seq 0 $cass_num); do
    echo "check ca cert for cassandra-$index"
    kubectl exec -it cassandra-$index -- openssl x509 -enddate -noout -in /etc/cassandra/pki/ca.crt
    echo "Check peer cert validity for cassandra-$index"
    kubectl exec -it cassandra-$index -- openssl x509 -enddate -noout -in /etc/cassandra/pki/tls.crt
done

Run the following command to update the permissions of the file where you saved the code snippet as added in Step 8 above:
```
chmod +x check-certs.sh
```

Run the following command to check the validity of the CA and Cassandra pods cert’s expiry:

./check-cert.sh

The following is the sample output:

check ca cert for cassandra-0
notAfter=Jun 18 04:57:08 2033 GMT
Check peer cert validity for cassandra-0
notAfter=Jun 20 04:57:24 2025 GMT
check ca cert for cassandra-1
notAfter=Jun 18 04:57:08 2033 GMT
Check peer cert validity for cassandra-1
notAfter=Jun 20 05:33:22 2025 GMT
check ca cert for cassandra-2
notAfter=Jun 18 04:57:08 2033 GMT
Check peer cert validity for cassandra-2
notAfter=Jun 20 05:45:42 2025 GMT

Documentation Index

Software Post-Upgrade Checks

1.0 Introduction

2.0 Post Upgrade Checks

2.1 Option 1 – Remote

2.2 Option 2 - Local

2.3 Option 3 - IPMI

2.4 Cluster Health Checks

2.5 Certificate Manager Verification