Amazon Elastic Kubernetes Service with Fortanix Confidential Computing Manager

Last Updated: May 11, 2023 08:47

AMAZON ELASTIC KUBERNETES SERVICE WITH FORTANIX CONFIDENTIAL COMPUTING MANAGER.pdf
(400 KB)

1.0 Introduction

This article describes how to set up an Elastic Kubernetes Service (EKS) cluster as worker nodes in Fortanix Confidential Computing Manager (CCM) and and run the Nitro application on the EKS cluster.

1.1 Prerequisites

The worker nodes of the EKS cluster should be of type i3en.xlarge or larger.
You must have EKS cluster created.

2.0 Running Pre-User Data Script

Run the following command to log in to each worker node using the ssh client connect:

ssh -i "psa-710-nitro.pem" ec2-user@ec2-3-101-65-194.us-west-1.compute.amazonaws.com

Run the pre_userdata.sh script on each worker nodes:

sudo amazon-linux-extras install aws-nitro-enclaves-cli -y
sudo yum install aws-nitro-enclaves-cli-devel -y
sudo usermod -aG ne ec2-user
sudo usermod -aG docker ec2-user
#sudo sed -i "s/cpu_count: 2/cpu_count: 4/g" /etc/nitro_enclaves/allocator.yaml
sudo sed -i "s/memory_mib: 512/memory_mib: 2560/g" /etc/nitro_enclaves/allocator.yaml
sudo systemctl start nitro-enclaves-allocator.service
sudo systemctl enable nitro-enclaves-allocator.service
sudo systemctl start docker
sudo systemctl enable docker
sudo chgrp ne /dev/nitro_enclaves
echo "Done with AWS Nitro enclave Setup"
sudo reboot

You must increase the memory of HughPages. Run the following commands to update the value of memory_mib parameter to 2560 in /etc/nitro_enclaves/allocator.yaml file:
```
---
# Enclave configuration file.
#
# How much memory to allocate for enclaves (in MiB).
memory_mib: 2560
#
# How many CPUs to reserve for enclaves.
cpu_count: 2
```
NOTE

Ensure that the total enclave memory required on a node is sum of 512 MB for em-agent node and Enclave memory your application requires. This memory is passed as part of Nitro app conversion.

Restart the worker node to reflect the changes:
```
reboot
```

3.0 Get Kubernetes Credentials

Run the following command to get kubeconfig:

eksctl utils write-kubeconfig --cluster=<cluster_name>

Run the following command to label all the nodes with Nitro Enclave OS capabilities:
```
kubectl label node <node-name> smarter-device-manager=enabled
```
NOTE

Ensure that you use private IP DNS name of the nodes.
Run the following command to label all the nodes with enclave.example.com/type=nitro:
```
kubectl label node <node-name> enclave.example.com/type=nitro
```
Run the following command to install the Smarter Device Manager on the Kubernetes cluster:
```
kubectl label node <node-name> enclave.example.com/type=nitro
```
NOTE

Ensure that you update the value for nummaxdevices parameter to a higher value than 1, such as 10.
Run the following command to install Smart Device Manager:
```
apply -f smarter-device-manager-ds-with-cm.yaml
```

4.0 Creating Secrets

Run the following command to create a secret to access the ECR registry:

kubectl create secret docker-registry regcred   --docker-server=513076507034.dkr.ecr.us-west-1.amazonaws.com   --docker-username=AWS --docker-password=$(aws ecr get-login-password)

Run the following command to create a secret for join token:

kubectl create secret generic em-token --from-literal=token=<join-token-from-account>

5.0 Installing Node Agent

Run the following command to install the node agent as daemonset:

kubectl apply -f agent-daemonset.yaml

The following is the content for agent-daemonset.yaml file:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: em-agent
  namespace: default
  labels:
    component: em-agent
spec:
  selector:
    matchLabels:
      component: em-agent
  template:
    metadata:
      labels:
        component: em-agent
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      volumes:
      - name: hugepage
        emptyDir:
          medium: HugePages
      - name: log
        hostPath:
          path: /var/log/nitro_enclaves
      - name: socket-path
        emptyDir: {}
      - name: node-data
        hostPath:
          path: /tmp/em-agent-nitro
      imagePullSecrets:
      - name: regcred
      containers:
      - name: em-agent
        image: "513076507034.dkr.ecr.us-west-1.amazonaws.com/em-agent-nitro:3.13.52"
        resources:
          limits:
            smarter-devices/nitro_enclaves: "1"
            hugepages-2Mi: 512Mi
            memory: 2Gi
            cpu: 250m
          requests:
            smarter-devices/nitro_enclaves: "1"
        volumeMounts:
        - mountPath: /dev/hugepages
          name: hugepage
          readOnly: false
        - name: log
          mountPath: /var/log/nitro_enclaves
        - name: socket-path
          mountPath: /run/nitro_enclaves
        - name: node-data
          mountPath: /tmp/em-agent-nitro
        ports:
        - containerPort: 9092
          name: http
          protocol: TCP
          hostPort: 9092
        env:
        - name: AGENT_MANAGER_AUTH_BASIC_TOKEN
          valueFrom:
            secretKeyRef:
              name: em-token
              key: token
        - name: MANAGER_ENDPOINT
          value: "ccm.test.fortanix.com"
        - name: MALBORK_LOG_DEBUG
          value: "true"
        - name: NODE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

NOTE

Update the image to the required em-agent version and MANAGER_ENDPOINT environment variable (test, stage, production) as per your requirement.
The em-agent images are available on the following paths:
- Unsigned for test: https://us-west-1.console.aws.amazon.com/ecr/repositories/private/513076507034/em-agent-nitro-unsigned?region=us-west-1
- Signed for stage and production: https://us-west-1.console.aws.amazon.com/ecr/repositories/private/513076507034/em-agent-nitro?region=us-west-1

6.0 Reinstalling Node Agent in a Different Account

This section lists the steps for installing the node agent in an account and switching it to different account.

Run the following command to delete /tmp/em-agent-nitro parameter from each worker nodes:

# log into the node through debug container.
$ kubectl debug node/<node-name> -it --image=busybox
# inside container now run chroot to access the node.
$ chroot /host bash
# Delete the file
$ rm -rf /tmp/em-agent-nitro
# Exit twice to get out of the container
$ exit

Run the following command to create a new em-token secret:

$ kubectl delete secret em-token
$ kubectl create secret generic em-token --from-literal=token=<join-token-from-account>

Run the following command to restart the em-agent secret:
```
kubectl rollout restart ds em-agent
```

7.0 Converting Nitro Enclave OS Application

Refer to the User's Guide: Create an Image to know the steps for creating application in Fortanix Confidential Computing Manager user interface.

7.1 Creating NGINX Deployment

The following is the content of app-development.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-pod
  labels:
    app: my-pod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-pod
  template:
    metadata:
      labels:
        app: my-pod
    spec:
      imagePullSecrets:
      - name: regcred
      containers:
      - name: nginx
        image: 513076507034.dkr.ecr.us-west-1.amazonaws.com/development-images/ccm-automation-input-images:hellowworld
        ports:
        - name: http
          containerPort: 80
        imagePullPolicy: Always
        securityContext:
          privileged: true
        resources:
          limits:
            smarter-devices/nitro_enclaves: "1"
            hugepages-1Gi: 2Gi
            memory: 2Gi
            cpu: 250m
          requests:
            smarter-devices/nitro_enclaves: "1"
            hugepages-1Gi: 2Gi
        volumeMounts:
        - mountPath: /dev/hugepages
          name: hugepage
          readOnly: false
        - name: log
          mountPath: /var/log/nitro_enclaves
        - name: socket-path
          mountPath: /run/nitro_enclaves
        env:
        - name: NODE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: NODE_AGENT
          value: http://$(NODE_IP):9092/v1/
        - name: RUST_LOG
          value: debug
      volumes:
      - name: hugepage
        emptyDir:
          medium: HugePages
      - name: log
        hostPath:
          path: /var/log/nitro_enclaves
      - name: socket-path
        emptyDir: {}

Update the container's image details to run any other image. Run the following command to deploy the application:

kubectl apply -f app-deployment.yaml