Backup and Restore

1.0 Introduction

This article describes the procedures for backing up and restoring the Fortanix Confidential Computing Manager (CCM) On-premises platform.

It covers both automatic and manual backup methods, along with the procedure for restoring the platform after a complete cluster reset or cluster failure using Medusa backups.

The procedures in this article use the kubectl Command Line Interface (CLI) for backup and restore operations. You can alternatively use any Kubernetes management tool of your choice, provided it offers equivalent functionality.

2.0 Backup

Fortanix CCM On-premises supports Cassandra database backups using Medusa.

Backups include:

Database (Cassandra)
Application bootstrapping data (ConfigMaps and Secrets)
Custom Resource (CR) file

NOTE
The backup process is considered complete only after the database, ConfigMap, Secret, and CR backups have all been successfully captured.

2.1 Prerequisites

Before performing a backup, ensure the following details are captured:

Kubernetes version of the Azure Kubernetes Service (AKS) cluster
Subnet, networking, and node configuration details
Versions of microservices and Fortanix Armor Kubernetes Operator

Run the following command to list deployment images:

kubectl get deployments -n $namepace-o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[ ].image

2.2 Database Backup - Automatic Backup Scheduling

Backups are stored in Azure Blob Storage using the Kubernetes secret (For example, secret: medusa-azure-key-test) configured in the CR file.

Automatic backups are scheduled using a Cron job based on the value specified in the schedule field.

Example:

The following section in the CR file contains the backup configuration:

backup:
  secret: medusa-azure-key-test
  bucketName: medusa
  prefix: platform
  schedule: 0 * * * *
  storageProvider:
    mode: azureBlobs

Where,

secret – Specifies the name of the Kubernetes secret used to access Azure Blob Storage. For example, medusa-azure-key-test.
bucketName – Specifies the name of the storage container where backups are stored. For example, medusa.
prefix – Specifies the logical folder or path within the storage container. For example, platform.
schedule – Specifies the cron expression that defines how frequently backups are triggered. For example, 0 * * * * (runs every hour).
storageProvider.mode – Specifies the storage provider used for backups. For example, azureBlobs.

Run the following command to view available backups:

kubectl get medusabackup -n $namespace

Here, the namespace refers to a Kubernetes namespace. For example, prod.

This command lists backup status, size, and completion details.

Example output:

NAME                   STARTED   FINISHED   NODES   FILES   SIZE       COMPLETED   STATUS
cassandra-1777460400   21h       21h        1                          3           SUCCESS
cassandra-1777464000   20h       20h        3       2998    21.69 MB   3           SUCCESS
cassandra-1777467600   19h       19h        3       3318    38.16 MB   3           SUCCESS

2.3 Database Backup - Manual

Manual backups can be triggered using a Medusa backup job.

Perform the following steps to create a manual database backup:

Create a file named backup.yaml with the following content:

apiVersion: medusa.k8ssandra.io/v1alpha1
kind: MedusaBackupJob
metadata:
  name: medusa-backup-manual-new1  # Name of the backup
  namespace: $namespace
  labels:
    cassandra.datastax.com/cluster: main  #By default
    cassandra.datastax.com/datacenter: dc1  #By default
spec:
  backupType: full
  cassandraDatacenter: dc1      #By default

Run the following command to trigger the backup:
```
kubectl apply -f backup.yaml -n $namespace
```

2.4 ConfigMap and Secret Backup

After completing the database backup, back up the application bootstrapping data.

Run the following command to back up the ConfigMap:
```
kubectl get configmap malbork-bootstrapping-data -n $namespace -o yaml | kubectl neat > malbork-bootstrapping-data.yaml
```
This command retrieves the ConfigMap from the cluster and saves it locally as a YAML file (malbork-bootstrapping-data.yaml).
Run the following command to back up the Secret:
```
kubectl get secret malbork-bootstrapping-secret-data -n $namespace -o yaml | kubectl neat > malbork-bootstrapping-secret-data.yaml
```
This command retrieves the Secret from the cluster and saves it locally as a YAML file (malbork-bootstrapping-secret-data.yaml).

2.5 CR Backup

Ensure that the cr.yaml file used in the previous deployment is available.

3.0 Restore

The restoration process uses Medusa backups to recover Cassandra data and restore platform services after a cluster failure or reset.

This procedure includes rebuilding the cluster (if required), pausing operator reconciliation, restoring Cassandra backups, applying the backed-up ConfigMaps and Secrets, and resuming the Fortanix Armor Kubernetes Operator to unpause the reconciliation, which brings the platform back to a fully operational state.

NOTE
Restoration is supported only on the same physical nodes (specifically, the same physical CPUs), and the number of nodes must also remain unchanged.
Fortanix CCM will not be accessible during the restoration process.
Restoring backups to a different Azure Kubernetes Service (AKS) cluster is not supported.

Restoration can be performed when Fortanix Armor Kubernetes Operator and microservices are deployed on the cluster.

If the cluster is lost, rebuild it using the same node configuration, then apply the backup CR by following the Installation Guide - On-premises.

NOTE
If the backup CR is lost and must be recreated on the same nodes, the system may not bootstrap completely, and some pods may fail to reach the Running state.

3.1 Prerequisites

Before starting the restoration process, ensure the following:

Only Medusa backups can be restored.
The cluster must be running the same versions of the operator, microservices, and Kubernetes that were in use when the backup was taken.
Ensure that the database backups, ConfigMap backups, Secret backups, and CR backups are available.

3.2 Restoration Procedure

Perform the following steps to reset and restore the cluster using a previously created backup.

NOTE
Before you begin, ensure that the system is fully deployed with all microservices and the Fortanix Armor Kubernetes Operator and Cassandra pods (for example, main-dc1-rack1-sts-0) are up and running.

In the cr.yaml file, under spec section, update the paused field to true to pause the Fortanix Armor Kubernetes Operator reconciliation:
```
paused: true   # true = pause operator reconciliation; false = resume operator reconciliation
```
Run the following command to apply the CR:
```
kubectl apply -f cr.yaml -n $namespace
```

Run the following commands to delete the Armor microservice deployments before starting the restoration process:

kubectl delete deployment alhambra bodrum ccm-frontend-X-Y-ZZZZ main-dc1-reaper malbork methoni microservice-certificate-controller -n $namespace

Run the following command to verify the available Medusa backups:
```
kubectl get medusabackup -n $namespace
```

Create a file named restore.yaml with the following content:

apiVersion: medusa.k8ssandra.io/v1alpha1
kind: MedusaRestoreJob
metadata:
  name: cassandra-1777881600 # A unique name for RESTORE job
  namespace: $namespace
spec:
  cassandraDatacenter: dc1
  backup: cassandra-1777881600 # backup file to be restored

Run the following command to apply the restore job and start the Cassandra restoration process:

kubectl apply -f restore.yaml -n $namespace

Run the following command to verify that the Medusa restore job has completed successfully:
```
kubectl get medusarestorejob -n $namespace
```
Ensure that the restore job has completed successfully and all Cassandra pods are in the Running state.
Run the following commands to restore the backed-up ConfigMap and Secret data:
NOTE
This step is required only if the Kubernetes cluster was recreated and the backed-up ConfigMap and Secret data must be restored. This step can be performed even when the Fortanix Armor Kubernetes Operator reconciliation is still paused.
```
kubectl apply -f malbork-bootstrapping-data.yaml -n $namespace
kubectl apply -f malbork-bootstrapping-secret-data.yaml -n $namespace
```
In the cr.yaml file, under spec section, update the paused field to false in the Fortanix Armor Kubernetes Operator CR and apply the cluster file to resume operator reconciliation:
```
paused: false
```
Wait until all platform pods are in the Running state.