Fortanix Data Security Manager Software Post-Upgrade Checks

1.0 Introduction

This article describes the checks to be performed by the System Administrator after upgrading the Fortanix DSM software.

2.0 Post Upgrade Checks

After the Fortanix DSM software is upgraded on your machine, perform the following steps to check if the cert-manager is upgraded successfully:

  1. Run the following command to check for all the resources in the cert-manager namespace:

    kubectl get all -n cert-manager

    The following is the sample output:

    NAME                                                       READY   STATUS    RESTARTS      AGE
    pod/cert-manager-csi-driver-hc7gr                          3/3     Running   4 (59m ago)   101m
    pod/certmanager-cert-manager-6c6bdd85d9-kzh7b              1/1     Running   0             63m
    pod/certmanager-cert-manager-cainjector-7b7cbc6988-6hp7d   1/1     Running   0             63m
    pod/certmanager-cert-manager-webhook-555cbb78cd-t6w2r      1/1     Running   0             63m
    
    NAME                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
    service/certmanager-cert-manager           ClusterIP   10.245.241.3    <none>        9402/TCP   101m
    service/certmanager-cert-manager-webhook   ClusterIP   10.245.190.11   <none>        443/TCP    101m
    
    NAME                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    daemonset.apps/cert-manager-csi-driver   1         1         1       1            1           <none>          101m
    
    NAME                                                  READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/certmanager-cert-manager              1/1     1            1           101m
    deployment.apps/certmanager-cert-manager-cainjector   1/1     1            1           101m
    deployment.apps/certmanager-cert-manager-webhook      1/1     1            1           101m
    
    NAME                                                             DESIRED   CURRENT   READY   AGE
    replicaset.apps/certmanager-cert-manager-6c6bdd85d9              1         1         1       101m
    replicaset.apps/certmanager-cert-manager-cainjector-7b7cbc6988   1         1         1       101m
    replicaset.apps/certmanager-cert-manager-webhook-555cbb78cd      1         1         1       101m
  2. Run the following command to view the helm chart status of cert-manager and check the version number:

    helm list -A

    The following is the sample output:

    NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                          APP VERSION
    certmanager     cert-manager    10              2023-05-30 14:27:59.266416986 +0000 UTC deployed        cert-manager-v1.11.2           v1.11.2
    csiplugin       cert-manager    10              2023-05-30 14:28:01.027483969 +0000 UTC deployed        cert-manager-csi-driver-v0.5.0 v0.5.0

    NOTE

    Ensure that the version number of cert-manager is 1.11.2.

  3. Run the following command to view the list of cert-manager pods:

    kubectl get pods -n cert-manager

    The following is the sample output:

    NAME                                                   READY   STATUS    RESTARTS      AGE
    cert-manager-csi-driver-hc7gr                          3/3     Running   4 (53m ago)   95m
    certmanager-cert-manager-6c6bdd85d9-kzh7b              1/1     Running   0             58m
    certmanager-cert-manager-cainjector-7b7cbc6988-6hp7d   1/1     Running   0             58m
    certmanager-cert-manager-webhook-555cbb78cd-t6w2r      1/1     Running   0             58m
    root@ip-172-31-0-42:/home/administrator#
  4. Run the following command to check the status of the Cassandra pods:

    kubectl get pods | grep cassandra

    The following is the sample output:

    cassandra-0 1/1 Running 0 57m
  5. Run the following command to check the value of DEPLOYMENT_STATUS variable in all the Cassandra pods:

    kubectl exec -it cassandra-0 -- env | grep DEPLOYMENT_STAGE
    DEPLOYMENT_STAGE=CERT_MANAGER_ONLY

    NOTE

    Ensure that the value is CERT_MANGER_ONLY.

  6. Run the following command to check if the configmap is created or not with the name cassandra-cert-manager-migration-state:

    kubectl get cm cassandra-cert-manager-migration-state
    NAME                                     DATA   AGE
    cassandra-cert-manager-migration-state   1      99m
    
    kubectl get cm cassandra-cert-manager-migration-state -ojsonpath='{.data}'
    {"DEPLOYMENT_STAGE":"CERT_MANAGER_ONLY"}
    
  7. Run the following command to check the secret resource with the name cassandra-mtls-ca:

    kubectl get secrets cassandra-mtls-ca

    The following is the sample output:

    NAME TYPE DATA AGE
    cassandra-mtls-ca kubernetes.io/tls 3 100m
  8. Save the following code snippet to a required file:

    #!/bin/bash
    cass_num=$(kubectl get statefulsets cassandra -ojsonpath='{.status.replicas}')
    cass_num=$((cass_num-1))
    for index in $(seq 0 $cass_num); do
        echo "check ca cert for cassandra-$index"
        kubectl exec -it cassandra-$index -- openssl x509 -enddate -noout -in /etc/cassandra/pki/ca.crt
        echo "Check peer cert validity for cassandra-$index"
        kubectl exec -it cassandra-$index -- openssl x509 -enddate -noout -in /etc/cassandra/pki/tls.crt
    done
  9. Run the following command to update the permissions of the file where you saved the code snippet as added in Step 8 above:

    chmod +x check-certs.sh
  10. Run the following command to check the validity of the CA and Cassandra pods cert’s expiry:

    ./check-cert.sh

    The following is the sample output:

    check ca cert for cassandra-0
    notAfter=Jun 18 04:57:08 2033 GMT
    Check peer cert validity for cassandra-0
    notAfter=Jun 20 04:57:24 2025 GMT
    check ca cert for cassandra-1
    notAfter=Jun 18 04:57:08 2033 GMT
    Check peer cert validity for cassandra-1
    notAfter=Jun 20 05:33:22 2025 GMT
    check ca cert for cassandra-2
    notAfter=Jun 18 04:57:08 2033 GMT
    Check peer cert validity for cassandra-2
    notAfter=Jun 20 05:45:42 2025 GMT