---
title: "(Release 4.31) Kubernetes Version Upgrade to 1.29 K8s"
slug: "fortanix-data-security-manager-release-431-kubernetes-version-upgrade-to-129-k8s"
updated: 2026-04-01T07:36:03Z
published: 2025-08-06T07:24:28Z
---

> ## Documentation Index
> Fetch the complete documentation index at: https://support.fortanix.com/llms.txt
> Use this file to discover all available pages before exploring further.

# (Release 4.31) Kubernetes Version Upgrade to 1.29 K8s

## 1.0 Introduction

The purpose of this article is to describe the steps to upgrade Kubernetes from version 1.27.06 to 1.29.6 for Fortanix-Data-Security-Manager (DSM) release 4.31.

## 2.0 Overview

The Fortanix DSM 4.31 release will upgrade the system from Kubernetes version 1.27 to 1.29. Subsequent Kubernetes upgrades will be released as part of regular upgrades or could continue to be independent upgrades.

After upgrading Fortanix DSM to the 4.31 version, you will not be able to downgrade to previous releases. The Fortanix DSM UI will not allow a downgrade after 4.23 is installed. Kindly work with Fortanix Support to ensure you have a valid backup that can be used to perform a manual recovery.

Also, you will need to upgrade Fortanix DSM to 4.23 before moving to any future release.

## 3.0 Pre-Upgrade Checks

Before upgrading the Kubernetes, ensure the following:

### 3.1 Check and Manage Disk Space

1. Run the following command to check if the disk space of more than 15 GB is available in `/var` and root (`/`) directories:

```bash
$ df -h /var/ /
```

The following is the sample output:

```bash
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 993G 29G 964G 3% /
/dev/nvme0n1p1 993G 29G 964G 3% /
```
2. Run the following command to delete the oldest version of Fortanix DSM from UI if the disk space is less than 15 GB:

```bash
$ df -h /var/ /
```

The following is the sample output:

```bash
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/main-var 47G 26G 21G 56% /var
/dev/sda2 47G 13G 33G 28% /
```

### 3.2 Configure and Validate Kubernetes

1. Verify the following keys in `kube-apiserver.yaml` of each node and ensure that the assigned IP address is same as the host IP.

In case of any mismatch, edit the yaml file to replace the assigned IP address with host IP.

The following lines are reference from `/etc/kubernetes/manifests/kube-apiserver.yaml` file.
  - `kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint`
  - `advertise-address`
  - `startupProbe IP`
  - `readinessProbe IP`
  - `livenessProbe IP`
  - Annotation:

```bash
annotations:
   kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 172.31.1.166:6443
```
  - Advertise-address:

```bash
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=172.31.1.166
```
  - livenessProbe:

```bash
livenessProbe:
     failureThreshold: 8 
     httpGet: 
       host: 172.31.1.166
```
  - ReadinessProbe:

```bash
readinessProbe:
     failureThreshold: 3
     httpGet:
       host: 172.31.1.166
```
  - startupProbe:

```bash
startupProbe:
     failureThreshold: 24
     httpGet:
       host: 172.31.1.166
```

### 3.3 Check Software Versions in Endpoints

1. Run the following command to check if all software versions are available in all the endpoints:

```bash
kubectl get ep -n swdist
```

The following is the sample output:

```bash
NAME      ENDPOINTS                                         AGE
swdist    10.244.0.212:22,10.244.1.191:22,10.244.2.152:22   242d
v2649     10.244.0.212:22,10.244.1.191:22,10.244.2.152:22   4d
v2657     10.244.0.212:22,10.244.1.191:22,10.244.2.152:22   2d
```
2. Run the following command to check the status of docker registry:

```bash
systemctl status docker-registry
```

Ensure that the status is active and running before and after the software is uploaded.

### 3.4 Check Cluster and Node Health

1. Run the following command to ensure that the overlay mount matches with this on each node:

```bash
cat /etc/systemd/system/var-opt-fortanix-swdist_overlay.mount.d/options.conf
[Mount]
```

```bash
Options=lowerdir=/var/opt/fortanix/swdist/data/vXXXX/registry:/var/opt/fortanix/swdist/data/vYYYY/registry
```

Here, ‘`vXXXX`’ is the previous version and ‘`vYYYY`’ is the upgraded version.
2. Ensure that the latest backup is triggered and verify that it is a successful backup (size and other metrics).
3. All nodes must report as healthy and be running Kubernetes version `1.27.9` and kernel `5.4.0-173-generic`. Run the following command to get the nodes and list the IP:

```bash
kubectl get nodes -o wide
```

Look for the version number under the column `VERSION` and it must be `v1.21.14` for each of the nodes.

```bash
NAME            STATUS ROLES        AGE   VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME 
ip-172-31-0-189 Ready control-plane 3h44m v1.27.6 172.31.0.189 <none> Ubuntu 20.04.6 LTS 5.4.0-173-generic containerd://1.7.12 
ip-172-31-1-110 Ready control-plane 3h37m v1.27.6 172.31.1.110 <none> Ubuntu 20.04.6 LTS 5.4.0-173-generic containerd://1.7.12 
ip-172-31-2-217 Ready control-plane 3h33m v1.27.6 172.31.2.217 <none> Ubuntu 20.04.6 LTS 5.4.0-173-generic containerd://1.7.12
```
4. All pods are healthy in the `default`, `swdist`and `kube-system` namespaces.
5. Run the following command to check `kubeadm` configuration on the cluster:

```bash
kubectl get configmap kubeadm-config -oyaml -nkube-system
```

This should return the following values for parameters in the master configuration:
  - kubernetesVersion: v1.27.6
  - imageRepository: *http://containers.fortanix.com:5000/*

### 3.5 Check Etcd Cluster and Component

1. Run the following command to check the status of `etcd` and if `isLeader=true` is assigned to one of the `etcd` node.
  - `etcd` should be TLS migrated.
2. Run the following command to generate the list of `etcd` members:

```bash
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl exec etcd-ip-172-31-0-84 -nkube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/healthcheck-client.crt --key /etc/kubernetes/pki/etcd/healthcheck-client.key --endpoints https://127.0.0.1:2379 member list
Defaulted container "etcd" out of: etcd, etcd-wait (init)
```

The following is the sample output of the above command:

```bash
66209b315d98f7a5, started, ip-172-31-0-84, https://172.31.0.84:2380, https://172.31.0.84:2379, false
```
3. Run the following command to ensure that the version of `etcd` on each of the `etcd` pods is `3.5.7`:

```bash
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl exec etcd-ip-172-31-0-84 -nkube-system -- etcd --version
Defaulted container "etcd" out of: etcd, etcd-wait (init)
```

The following is the sample output of the above command:

```bash
etcd Version: 3.5.7
Git SHA: 215b53cf3
Go Version: go1.17.13
Go OS/Arch: linux/amd64
```
4. Run the following command to check the health of `etcd` cluster and ensure that the health of the cluster is healthy:

```bash
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl exec etcd-ip-172-31-0-84 -nkube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/healthcheck-client.crt --key /etc/kubernetes/pki/etcd/healthcheck-client.key --endpoints https://127.0.0.1:2379 endpoint health
Defaulted container "etcd" out of: etcd, etcd-wait (init)
```

The following is the sample output of the above command:

```bash
https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 7.286578ms
```
5. On each node, navigate to `/etc/kubernetes/manifests` directory and run the following command to check the image versions for all Kubernetes control-plane components:

```bash
cd /etc/kubernetes/manifests/
ls
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml
cat etcd.yaml | grep "image:"
    image: containers.fortanix.com:5000/etcd:3.5.7-0
    image: containers.fortanix.com:5000/etcd:3.5.7-0
cat kube-apiserver.yaml | grep "image: "
    image: containers.fortanix.com:5000/kube-apiserver:v1.27.6
cat kube-controller-manager.yaml | grep "image: "
    image: containers.fortanix.com:5000/kube-controller-manager:v1.27.6
cat kube-scheduler.yaml | grep "image: "
    image: containers.fortanix.com:5000/kube-scheduler:v1.27.6
```
6. Perform the following steps to check the expiry of the Kubernetes certificates.
  1. Check the certificates under `/etc/kubernetes/pki` and `/etc/kubernetes/pki/etcd` directories.
  2. Run the following command to renew the expired certificates:

```bash
/opt/fortanix/sdkms/bin/renew-k8s-certs.sh
```
7. Run the following command on each node to check the status of `kubelet`, `docker`, and `docker-registry` service:

```bash
systemctl status containerd
systemctl status kubelet
systemctl status docker-registry
```

> [!NOTE]
> NOTE
> 
> Ensure that the status of the services is `Running`.

## 4.0 Post-Upgrade Checks

Ensure to refer to[Pre-Upgrade Checks](/v1/docs/fortanix-data-security-manager-release-423-kubernetes-version-upgrade-to-127-k8s#30-preupgrade-checks) before upgrading the Kubernetes:

### 4.1 Check Node and Deployment Status

1. Run the following command to check the status of the deploy job:

```bash
# kubectl get pods | grep deploy
```

The following is the sample output of the above command:

```bash
deploy-vqq7r     0/1     Completed   0    32m
```

> [!NOTE]
> NOTE
> 
> Ensure that the status of the pod is `Completed`.
2. Run the following command to get the list of the deploy job:

```bash
# kubectl get job deploy
```

The following is the sample output of the above command:

```bash
NAME     COMPLETIONS   DURATION   AGE
deploy   1/1           76m      91d
```

> [!NOTE]
> NOTE
> 
> Verify the completion and duration of the job.
3. If you are using [DC Labeling](/v1/docs/fortanix-data-security-manager-data-center-labeling), run the following command to verify if the zone label is added by the YAML of the node:

```bash
kubectl get node node_name -o yaml | grep -i 'zone'
```
4. Run the following command to check the status of the nodes and the k8s version and the role must be control-plane:

```bash
kubectl get nodes -o wide
```

The following is the sample output of the above command:

```bash
NAME            STATUS ROLES         AGE  VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME 
ip-172-31-0-189 Ready control-plane 3h44m v1.29.6 172.31.0.189 <none> Ubuntu 20.04.6 LTS 5.4.0-190-generic containerd://1.7.12 
ip-172-31-1-110 Ready control-plane 3h37m v1.29.6 172.31.1.110 <none> Ubuntu 20.04.6 LTS 5.4.0-190-generic containerd://1.7.12 
ip-172-31-2-217 Ready control-plane 3h33m v1.29.6 172.31.2.217 <none> Ubuntu 20.04.6 LTS 5.4.0-190-generic containerd://1.7.12
```

> [!NOTE]
> NOTE
> 
> Ensure the following:
> 
> - Status of the nodes is `Ready`
> - VERSION column reflects `v1.29.6`
> - ROLE column reflects `control-plane`
> - KERNEL_VERSION reflects `5.4.0-190-generic`

### 4.2 Check Kubernetes and Component Version

1. Run the following command to generate the list of `etcd` members:

```bash
kubectl exec etcd-ip-172-31-0-189 -nkube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/healthcheck-client.crt --key /etc/kubernetes/pki/etcd/healthcheck-client.key --endpoints https://127.0.0.1:2379 member list
Defaulted container "etcd" out of: etcd, etcd-wait (init)
```

The following is the sample output of the above command:

```bash
189e28c15b97e101, started, ip-172-31-1-110, https://172.31.1.110:2380, https://172.31.1.110:2379, false 
196b7e4e172e3774, started, ip-172-31-0-189, https://172.31.0.189:2380, https://172.31.0.189:2379, false 
5bf2ddea0b074725, started, ip-172-31-2-217, https://172.31.2.217:2380, https://172.31.2.217:2379, false
```
2. Run the following command to check if `kube-proxy` is upgraded to image `v1.29.6-1-11b078244f45f2`:

```bash
$ sudo -E kubectl describe ds kube-proxy -nkube-system | grep Image
```

The following is the sample output of the above command:

```plaintext
Image: containers.fortanix.com:5000/kube-proxy:v1.27.6-1-840fae1b914b0d
```
3. Run the following command to check if `kured` pod is running with image version `1.15.1`:

```bash
$ sudo -E kubectl describe ds kured -nkube-system | grep Image
```

The following is the sample output of the above command:

```bash
Image: containers.fortanix.com:5000/kured:1.15.1
```
4. Run the following command on each of the nodes in the cluster to check if `kube-apiserver`, `kube-controller-manager`, `kube-scheduler` are upgraded to `1.27.6`:

```bash
$ sudo cat /etc/kubernetes/manifests/kube-scheduler.yaml | grep "image:"
    image: containers.fortanix.com:5000/kube-scheduler:v1.29.6
$ sudo cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep "image:"
    image: containers.fortanix.com:5000/kube-controller-manager:v1.29.6
$ sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep "image:"
    image: containers.fortanix.com:5000/kube-apiserver:v1.29.6
```
5. Run the following command to check the version of `etcd`:

```bash
kubectl get pod etcd-ip-172-31-0-189 -n kube-system -o yaml | grep image:
```

The following is the sample output of the above command:

```bash
image: containers.fortanix.com:5000/etcd:3.5.12-0
image: containers.fortanix.com:5000/etcd:3.5.12-0
image: containers.fortanix.com:5000/etcd:3.5.12-0
image: containers.fortanix.com:5000/etcd:3.5.12-0
```
6. Run the following command to check the version of `cert-manager` helm chart:

```bash
helm list -A
```

The following is the sample output of the above command:

```bash
NAME        NAMESPACE    REVISION UPDATED    STATUS             CHART APP          VERSION 
certmanager cert-manager 2        2024-07-10 11:57:19.980606498 +0000 UTC deployed cert-manager-v1.14.4 v1.14.4 
csiplugin   cert-manager 2        2024-07-10 11:57:22.410910496 +0000 UTC deployed cert-manager-csi-driver-v0.8.0 v0.8.0
```

> [!NOTE]
> NOTE
> 
> Ensure that the `helm chart` version is `1.14.4` and `csiplugin` version is `0.8.0`.
7. Run the following command to check if the Kubernetes version is upgraded to `v1.29.6` (including `kubeadm`, `kubectl`, `kubelet packages`):

```bash
$ dpkg -l | grep kube
```

The following is the sample output of the above command:

```bash
ii kubeadm 1.29.6-1.1fortanix amd64 Kubernetes Cluster Bootstrapping Tool
ii kubectl 1.29.6-1.1 amd64 Kubernetes Command Line Tool
ii kubelet 1.29.6-1.1 amd64 Kubernetes Node Agent
ii kubernetes-cni 1.2.0-00 amd64 Kubernetes CNI
```
8. Run the following command to check if image tag `0.25.0` for `swdist` container is updated:

```bash
$ sudo -E kubectl describe ds swdist -nswdist | grep Image
```

The following is the sample output of the above command:

```bash
    Image:      containers.fortanix.com:5000/swdist:0.28.0
    Image:      containers.fortanix.com:5000/swdist:0.28.0
    Image:      containers.fortanix.com:5000/swdist:0.28.0
    Image:      containers.fortanix.com:5000/swdist:0.28.0
    Image:      containers.fortanix.com:5000/swdist:0.28.0
    Image:      containers.fortanix.com:5000/swdist:0.28.0
    Image:      containers.fortanix.com:5000/swdist:0.28.0
    Image:      containers.fortanix.com:5000/swdist:0.28.0
```
9. Run the following command to check the replicas of `coredns` deployment:

```bash
sudo -E kubectl get pods -nkube-system -owide | grep coredns
```

The following is the sample output of the above command:

```bash
coredns-786bdcfc9c-bvzzf                  1/1     Running   0              30m   10.244.0.76   ip-172-31-0-189     <none> <none> 
coredns-786bdcfc9c-fkw7s                  1/1     Running   0              30m   10.244.1.76   ip-172-31-2-110     <none> <none> 
coredns-786bdcfc9c-r2s8c                  1/1     Running   0              30m   10.244.2.76    ip-172-31-1-217    <none> <none>        
```

> [!NOTE]
> NOTE
> 
> Ensure that number of duplicate `coredns` must be equal to the number of nodes in the cluster.
10. Run the following command to check the version of `flannel` and `flannel-plugin`:

```bash
kubectl get ds kube-flannel-ds -n kube-system -o yaml | grep image:
```

The following is the sample output of the above command:

```bash
image: containers.fortanix.com:5000/flannel:v0.25.1
image: containers.fortanix.com:5000/flannel-cni-plugin:v1.4.1flannel1
image: containers.fortanix.com:5000/flannel:v0.25.1
```

> [!NOTE]
> NOTE
> 
> Ensure that the `flannel` version is `0.25.1` and `flannel plugin` version is `1.4.1flannel1`.

### 4.3 Check cert-manager Configuration

1. Run the following command to check all the resources of `cert-manager`:

```bash
kubectl get all -n cert-manager
```

The following is the sample output of the above command:

```bash
NAME READY STATUS RESTARTS AGE
pod/cert-manager-csi-driver-9lvw2 3/3 Running 4 (14h ago) 15h
pod/certmanager-cert-manager-5fd9f859bb-7slz2 1/1 Running 0 14h
pod/certmanager-cert-manager-cainjector-5998546469-pk9kb 1/1 Running 0 14h
pod/certmanager-cert-manager-webhook-878f95fb5-699lp 1/1 Running 0 14h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/certmanager-cert-manager ClusterIP 10.245.213.126  <none> 9402/TCP 15h
service/certmanager-cert-manager-webhook ClusterIP 10.245.20.237  <none> 443/TCP 15h

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/cert-manager-csi-driver 1 1 1 1 1  <none> 15h

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/certmanager-cert-manager 1/1 1 1 15h
deployment.apps/certmanager-cert-manager-cainjector 1/1 1 1 15h
deployment.apps/certmanager-cert-manager-webhook 1/1 1 1 15h

NAME DESIRED CURRENT READY AGE
replicaset.apps/certmanager-cert-manager-5fd9f859bb 1 1 1 14h
replicaset.apps/certmanager-cert-manager-6c6bdd85d9 0 0 0 15h
replicaset.apps/certmanager-cert-manager-cainjector-5998546469 1 1 1 14h
replicaset.apps/certmanager-cert-manager-cainjector-7b7cbc6988 0 0 0 15h
replicaset.apps/certmanager-cert-manager-webhook-555cbb78cd 0 0 0 15h
replicaset.apps/certmanager-cert-manager-webhook-878f95fb5 1 1 1 14h
```
2. Run the following command to check the `DEPLOYMENT_STATUS` environment variable in all Cassandra pod. It should be set as `CERT_MANAGER_ONLY` as illustrated in the example for `cassandra-0`:

```bash
kubectl exec -it cassandra-0 -- env | grep DEPLOYMENT_STAGE
DEPLOYMENT_STAGE=CERT_MANAGER_ONLY
```
3. Run the following command to check the configmap with name `cassandra-cert-manager-migration-state`:

```bash
kubectl get cm cassandra-cert-manager-migration-state -ojsonpath='{.data}'
{"DEPLOYMENT_STAGE":"CERT_MANAGER_ONLY"}
```

## 5.0 Troubleshooting

1. In case kubelet client certificates expire (`/var/lib/kubelet/pki/kubelet-client.crt`) and there is no `/var/lib/kubelet/pki/kubelet-client-current.pem` file present, then you can create the certificates using the following commands:

```bash
TEMP_DIR=/etc/kubernetes/tmp
mkdir -p $TEMP_DIR
BACKUP_PEM="/var/lib/kubelet/pki/kubelet-client-current.pem"
KEY="/var/lib/kubelet/pki/kubelet-client.key"
CERT="/var/lib/kubelet/pki/kubelet-client.crt"

echo "Stopping kubelet service"
systemctl stop kubelet

echo "Creating a new key and cert file for kubelet auth"
nodename=$(echo "$HOSTNAME" | awk '{print tolower($0)}')
openssl req -out $TEMP_DIR/tmp.csr -new -newkey rsa:2048 -nodes -keyout $TEMP_DIR/tmp.key -subj "/O=system:nodes/CN=system:node:$nodename"
cat > $TEMP_DIR/kubelet-client.ext << HERE
keyUsage = critical,digitalSignature,keyEncipherment
extendedKeyUsage = clientAuth
HERE
echo "Signing the generated csr with kubernetes CA"
openssl x509 -req -days 365 -in $TEMP_DIR/tmp.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out $TEMP_DIR/tmp.crt -sha256 -extfile $TEMP_DIR/kubelet-client.ext
cp $TEMP_DIR/tmp.crt $CERT
cp $TEMP_DIR/tmp.key $KEY

chmod 644 $CERT
chmod 600 $KEY

if grep -q "client-certificate-data" $KUBELET_CONF; then
    echo "Updating file $KUBELET_CONF to add reference to restored certificates"
    sed -i "s|\(client-certificate-data:\s*\).*\$|client-certificate: $CERT|" $KUBELET_CONF
    sed -i "s|\(client-key-data:\s*\).*\$|client-key: $KEY|" $KUBELET_CONF
fi

echo "Starting kubelet service"
systemctl start kubelet
```
2. Upgrade on two node cluster can fail due to `etcd` quorum failure. In such a scenario, if pods are healthy, you can re-run the deploy job manually using the following command. This will eventually upgrade the cluster to `1.29.6`.

```bash
sdkms-cluster deploy --stage DEPLOY --version <version>
```

> [!WARNING]
> WARNING
> 
> Two node upgrades are not recommended.

Fortanix Data Security Manager (DSM) is the world’s first cloud service secured with Intel® SGX. With Fortanix DSM, you can securely generate, store, and use cryptographic keys and certificates, as well as other secrets such as passwords, API keys, tokens, or any blob of data. Your business-critical applications and containers can integrate with Fortanix DSM using legacy cryptographic interfaces (PKCS#11, CNG, and JCE) or using the native Fortanix DSM RESTful interface.