Kubernetes Root CA Rotation

  • Published on Jun 20, 2024
  • 3 minute(s) read

1.0 Introduction

The purpose of this guide is to describe the steps to manually rotate the Kubernetes Certificate Authority (CA) Certificates.

2.0 Prerequisites

  • Kubernetes Cluster and kubectl command-line tool to communicate with your cluster

  • Kubernetes server version >=v1.13

3.0 Kubernetes CA Rotation

The Kubernetes CA rotation script is present in the Fortanix DSM node at the location: /opt/fortanix/sdkms/bin.

The Kubernetes Root CA Certificates is used by the following files and entities:

  • admin.conf - For signing the client certificate

  • scheduler.conf - For signing the client certificate

  • controller-manager.conf - For signing the client certificate

  • kubelet.conf - For signing the client certificate

  • Cassandra - For serving as the root CA for Cassandra

  • apiserver - For certificate signing

  • apiserver-kubelet-client - For certificate signing

3.1 Steps to Run the Rotation Script

The following are the steps to run a rotation script assuming a 3-node cluster:

  1. Run the Kubernetes CA rotation script on the first node:

    KUBECONFIG=/etc/kubernetes/admin.conf ./k8s-ca-rotate.sh --firstnode

    • This step will back up the existing data present in /etc/kubernetes to /var/log/Fortanix.

    • It will create the following new files:

      • /etc/kubernetes/ca-new.key

      • /etc/kubernetes/ca-new.crt

    • You can also specify the size of the key to be created using the parameter --keylength. Allowed values are 2048|3072|4096.
      For example

      KUBECONFIG=/etc/kubernetes/admin.conf ./k8s-ca-rotate.sh --firstnode --keylength=4096

    • A new file /etc/kubernetes/ca-bundle.crt is created which is the combination of old (ca-old.crt) and new certificates (ca-new.crt).

    • A new file /etc/kubernetes/ca-bundle.key is created which is the combination of old (ca-old.key) and new certificates (ca-new.key).

    • A ConfigMap ca-rotation is created and mounted to Cassandra to keep the old and new certificates available until the CA rotation is completed.

  2. Run the Kubernetes CA rotation script again on the same node with the same parameters:

    KUBECONFIG=/etc/kubernetes/admin.conf ./k8s-ca-rotate.sh --firstnode

    • This will update the following files in /etc/kubernetes to include the CA bundle with /etc/kubernetes/pki/ca.crt:

      • admin.conf

      • scheduler.conf

      • controller-manager.conf

      • kubelet.conf (and /var/lib/kubelet/pki/kubelet-client-current.pem if present )

    • The following files are updated in /etc/kubernetes/pki and the new certs are signed by the new root CA /etc/kubernetes/ca.crt

      • apiserver

      • apiserver-kubelet-client

    • All daemon sets and deployments are patched to restart and use the new secret tokens signed by the CA bundle.

    • The cluster-info ConfigMap is updated to use the new CA for token generation for kubeadm join.

    • The following files in /etc/kubernetes/manifests are updated to include the CA bundle.

      • kube-controller-manager.yaml

      • kube-apiserver.yaml

    • The file /var/lib/kubelet/config.yaml is updated to include the CA bundle.

  3. Now copy /etc/kubernetes/ca* to the second node.

  4. Run the Kubernetes CA rotation script on the second node.

    KUBECONFIG=/etc/kubernetes/admin.conf ./k8s-ca-rotate.sh

  5. Next, copy /etc/kubernetes/ca* to the third node.

  6. Run the Kubernetes CA rotation script on the third node.

    KUBECONFIG=/etc/kubernetes/admin.conf ./k8s-ca-rotate.sh

  7. Run the Kubernetes CA rotation script on each node one by one again to sign the apiserver and apiserver-kubelet-client.

    sudo KUBECONFIG=/etc/kubernetes/admin.conf ./k8s-ca-rotate.sh --mid-rotate

    • The following files are updated in /etc/kubernetes/pki, and new certs are signed by the new root CA in /etc/kubernetes/pki/ca.crt

      • apiserver

      • apiserver-kubelet-client

  8. Run the Kubernetes CA rotation script again on the first node for post-rotation steps.

    KUBECONFIG=/etc/kubernetes/admin.conf ./k8s-ca-rotate.sh --post-rotate

    • This will update the following files in /etc/kubernetes to replace the CA bundle with /etc/kubernetes/ca.crt:

      • admin.conf

      • scheduler.conf

      • controller-manager.conf

      • kubelet.conf

    • The following files in /etc/kubernetes/manifests are updated to replace the CA bundle with /etc/kubernetes/pki/ca.crt:

      • kube-controller-manager.yaml

      • kube-apiserver.yaml

    • The file /var/lib/kubelet/config.yaml is updated to replace the CA bundle with /etc/kubernetes/ca.crt.

    • The file /etc/kubernetes/pki/ca.crt is renamed to /etc/kubernetes/pki/ca-old.crt.

    • The file/etc/kubernetes/pki/ca.key is renamed to /etc/kubernetes/pki/ca-old.key.

    • The file /etc/kubernetes/pki/ca-new.crt is renamed to /etc/kubernetes/pki/ca.crt.

    • The file/etc/kubernetes/pki/ca-new.key is renamed to /etc/kubernetes/pki/ca.key.

  9. Run the Kubernetes CA rotation script again on the second node with the post rotate flag.

    KUBECONFIG=/etc/kubernetes/admin.conf ./k8s-ca-rotate.sh --post-rotate

    • The file /etc/kubernetes/pki/ca.crt is renamed to /etc/kubernetes/pki/ca-old.crt.

    • The file/etc/kubernetes/pki/ca.key is renamed to /etc/kubernetes/pki/ca-old.key.

    • The file /etc/kubernetes/pki/ca-new.crt is renamed to /etc/kubernetes/pki/ca.crt.

    • The file/etc/kubernetes/pki/ca-new.key is renamed to /etc/kubernetes/pki/ca.key.

  10. Run the Kubernetes CA rotation script again on the third node with the post rotate flag and last node flag.

    KUBECONFIG=/etc/kubernetes/admin.conf ./k8s-ca-rotate.sh --post-rotate --lastnode

    • This will deploy Cassandra chart again.

    • The file /etc/kubernetes/pki/ca.crt is renamed to /etc/kubernetes/pki/ca-old.crt.

    • The file/etc/kubernetes/pki/ca.key is renamed to /etc/kubernetes/pki/ca-old.key.

    • The file /etc/kubernetes/pki/ca-new.crt is renamed to /etc/kubernetes/pki/ca.crt.

    • The file/etc/kubernetes/pki/ca-new.key is renamed to /etc/kubernetes/pki/ca.key.

  11. For Fortanix DSM versions below 4.19, you need to roll out the Cassandra statefulset to make use of the new ca.crt.

    sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl rollout restart statefulset cassandra

  12. Run the Kubernetes CA rotation script one by one on each node for clean-up.

    KUBECONFIG=/etc/kubernetes/admin.conf ./k8s-ca-rotate.sh --cleanup

    This will remove the backup of older /etc/kubernetes files.

3.2 Verify CA Rotation

  • Verify the new certs in etc/kubernetes/pki/ca.crt signed by the new key /etc/kubernetes/pki/ca.key.

  • Verify the new certs /etc/kubernetes/pki/apiserver.crt and /etc/kubernetes/pki/apiserver-kubelet-client.crt.

  • Verify the new certs in /etc/kubernetes/*.conf files (admin.conf, scheduler.conf, controller-manager.conf, and kubelet.conf).

  • Verify the new certs in /var/lib/kubelet/pki/kubelet-current-client.pem.

  • Verify the new certs in the Cassandra container (mounted from the host).

  • The file /etc/kubernetes/pki/ca.rotated should be present on the nodes. For subsequent rotations later, please delete the file.

Related articles