Software Post-Upgrade Checks

1.0 Introduction

This article describes the checks to be performed by the System Administrator after upgrading the Fortanix-Data-Security-Manager (DSM) software.

2.0 Post Upgrade Checks

After the Fortanix DSM software is upgraded on your machine, perform the following steps to check if cert-manager is upgraded successfully:

Run the following command to check for all the resources in the cert-manager namespace:

kubectl get all -n cert-manager

The following is the sample output:

NAME                                                       READY   STATUS    RESTARTS      AGE
pod/cert-manager-csi-driver-hc7gr                          3/3     Running   4 (59m ago)   101m
pod/certmanager-cert-manager-6c6bdd85d9-kzh7b              1/1     Running   0             63m
pod/certmanager-cert-manager-cainjector-7b7cbc6988-6hp7d   1/1     Running   0             63m
pod/certmanager-cert-manager-webhook-555cbb78cd-t6w2r      1/1     Running   0             63m

NAME                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/certmanager-cert-manager           ClusterIP   10.245.241.3    <none>        9402/TCP   101m
service/certmanager-cert-manager-webhook   ClusterIP   10.245.190.11   <none>        443/TCP    101m

NAME                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/cert-manager-csi-driver   1         1         1       1            1           <none>          101m

NAME                                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/certmanager-cert-manager              1/1     1            1           101m
deployment.apps/certmanager-cert-manager-cainjector   1/1     1            1           101m
deployment.apps/certmanager-cert-manager-webhook      1/1     1            1           101m

NAME                                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/certmanager-cert-manager-6c6bdd85d9              1         1         1       101m
replicaset.apps/certmanager-cert-manager-cainjector-7b7cbc6988   1         1         1       101m
replicaset.apps/certmanager-cert-manager-webhook-555cbb78cd      1         1         1       101m

Run the following command to view the helm chart status of cert-manager and check the version number:

helm list -A

The following is the sample output:

NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                          APP VERSION
certmanager     cert-manager    10              2023-05-30 14:27:59.266416986 +0000 UTC deployed        cert-manager-v1.11.2           v1.11.2
csiplugin       cert-manager    10              2023-05-30 14:28:01.027483969 +0000 UTC deployed        cert-manager-csi-driver-v0.5.0 v0.5.0

NOTE
Ensure that the version number of cert-manager is 1.11.2.

Run the following command to view the list of cert-manager pods:

kubectl get pods -n cert-manager

The following is the sample output:

NAME                                                   READY   STATUS    RESTARTS      AGE
cert-manager-csi-driver-hc7gr                          3/3     Running   4 (53m ago)   95m
certmanager-cert-manager-6c6bdd85d9-kzh7b              1/1     Running   0             58m
certmanager-cert-manager-cainjector-7b7cbc6988-6hp7d   1/1     Running   0             58m
certmanager-cert-manager-webhook-555cbb78cd-t6w2r      1/1     Running   0             58m
root@ip-172-31-0-42:/home/administrator#

Run the following command to check the status of the Cassandra pods:
```
kubectl get pods | grep cassandra
```
The following is the sample output:
```
cassandra-0 1/1 Running 0 57m
```
Run the following command to check the value of DEPLOYMENT_STATUS variable in all the Cassandra pods:
```
kubectl exec -it cassandra-0 -- env | grep DEPLOYMENT_STAGE
DEPLOYMENT_STAGE=CERT_MANAGER_ONLY
```
NOTE
Ensure that the value is CERT_MANGER_ONLY.

Run the following command to check if the configmap is created or not with the name cassandra-cert-manager-migration-state:

kubectl get cm cassandra-cert-manager-migration-state
NAME                                     DATA   AGE
cassandra-cert-manager-migration-state   1      99m

kubectl get cm cassandra-cert-manager-migration-state -ojsonpath='{.data}'
{"DEPLOYMENT_STAGE":"CERT_MANAGER_ONLY"}

Run the following command to check the secret resource with the name cassandra-mtls-ca:
```
kubectl get secrets cassandra-mtls-ca
```
The following is the sample output:
```
NAME TYPE DATA AGE
cassandra-mtls-ca kubernetes.io/tls 3 100m
```

Save the following code snippet to a required file:

#!/bin/bash
cass_num=$(kubectl get statefulsets cassandra -ojsonpath='{.status.replicas}')
cass_num=$((cass_num-1))
for index in $(seq 0 $cass_num); do
    echo "check ca cert for cassandra-$index"
    kubectl exec -it cassandra-$index -- openssl x509 -enddate -noout -in /etc/cassandra/pki/ca.crt
    echo "Check peer cert validity for cassandra-$index"
    kubectl exec -it cassandra-$index -- openssl x509 -enddate -noout -in /etc/cassandra/pki/tls.crt
done

Run the following command to update the permissions of the file where you saved the code snippet as added in Step 8 above:
```
chmod +x check-certs.sh
```

Run the following command to check the validity of the CA and Cassandra pods cert’s expiry:

./check-cert.sh

The following is the sample output:

check ca cert for cassandra-0
notAfter=Jun 18 04:57:08 2033 GMT
Check peer cert validity for cassandra-0
notAfter=Jun 20 04:57:24 2025 GMT
check ca cert for cassandra-1
notAfter=Jun 18 04:57:08 2033 GMT
Check peer cert validity for cassandra-1
notAfter=Jun 20 05:33:22 2025 GMT
check ca cert for cassandra-2
notAfter=Jun 18 04:57:08 2033 GMT
Check peer cert validity for cassandra-2
notAfter=Jun 20 05:45:42 2025 GMT