Amazon Elastic Kubernetes Service with Fortanix Confidential Computing Manager

1.0 Introduction

This article describes how to configure an Amazon Elastic Kubernetes Service (EKS) cluster and enroll its worker nodes in Fortanix Confidential Computing Manager (CCM).

2.0 Prerequisites

Ensure that you meet the following requirements:

You must have an active AWS subscription.
The EKS cluster worker nodes must use i3en.xlarge instance types or larger.
An Amazon EKS cluster must already be created.

3.0 Set Up Amazon EKS Cluster

Ensure that the Amazon EKS cluster and worker nodes are configured with the Nitro Enclaves Kubernetes device plugin. For more information on the procedures and steps, refer to the AWS official documentation.

You must increase the HugePages memory allocation by updating the value of the memory_mib parameter to 2560 in the /etc/nitro_enclaves/allocator.yaml file.

---
# Enclave configuration file.
#
# How much memory to allocate for enclaves (in MiB).
memory_mib: 2560
#
# How many CPUs to reserve for enclaves.
cpu_count: 2

NOTE
Ensure that a minimum of 512 MB of enclave memory is allocated for the em-agent node agent on each node, in addition to the enclave memory required for the application workload.

Run the following command to restart the worker node and apply the configuration changes:

reboot

4.0 Get Kubernetes Credentials

Perform the following steps to obtain Kubernetes credentials (kubeconfig) for cluster management:

Run the following command to get kubeconfig:

eksctl utils write-kubeconfig --cluster=<cluster_name>

aws eks --region us-west-1 update-kubeconfig --name <cluster-name>

Run the following command to label all the nodes with Nitro Enclave OS capabilities:
```
kubectl label node <node-name> smarter-device-manager=enabled
```
NOTE
Ensure that you use private IP Domain Name System (DNS) name of the nodes.
Run the following command to label all the nodes with enclave.example.com/type=nitro:
```
kubectl label node <node-name> enclave.example.com/type=nitro
```
Run the following command to install the Smarter Device Manager on the Kubernetes cluster:
```
kubectl apply -f smarter-device-manager-ds-with-cm.yaml
```
NOTE
The Smarter Device Manager YAML configuration file is available in smarter-device-manager-ds-with-cm.yaml. Ensure that the value of the nummaxdevices parameter is updated to a value greater than 1, for example, 10.

5.0 Provide AWS CCM User Access to Your EKS Cluster

Perform the following steps to provide AWS CCM user access to the EKS cluster:

Run the following command to update the aws-auth configmap:
```
 kubectl edit -n kube-system configmap/aws-auth
```

Add the following changes into the configmap file:

mapUsers: |
    - userarn: arn:aws:iam::513076507034:user/em-admin@fortanix.com
    username: em-admin@fortanix.com
    groups:
    - system:masters

This configuration assumes that the credentials associated with em-admin@fortanix.com are configured as eks_config in Fortanix CCM clusters.

6.0 Create Kubernetes Secrets

Perform the following steps to create a Kubernetes secret for the cluster :

Run the following command to create a secret to access the ECR registry:

kubectl create secret docker-registry regcred   --docker-server=513076507034.dkr.ecr.us-west-1.amazonaws.com   --docker-username=AWS --docker-password=$(aws ecr get-login-password)

NOTE
Skip this step if the image is stored in a public repository.

Retrieve the join token for your Fortanix CCM account from the CCM user interface (UI) and store it as a Kubernetes secret in your cluster. To generate your join token, log in to https://ccm.fortanix.com/ the URL https://armor.fortanix.com/ if you are using the Armor SaaS deployment. For on-premises deployments, navigate to the configured Armor domain (https://<armor-domain>).
1. In the CCM UI left navigation panel, navigate to Infrastructure → COMPUTE NODES → Intel SGX, and then click ADD NODE.
  Figure 1: Add compute node
2. In the Enroll Compute Node window, click COPY to copy the join token. This Join Token is used by the compute node to authenticate itself.
3. Run the following command to store the join token as a Kubernetes secret for the cluster. Replace the <join-token-from-account> value below with your token.
```
kubectl create secret generic em-token --from-literal=token=<join-token-from-account>
```

7.0 Install Fortanix CCM Node Agent

Perform the following steps to install the node agent on your Kubernetes cluster:

Run the following command to deploy the node agent as a daemonset:

kubectl apply -f agent-daemonset.yaml

The following is the content for the agent-daemonset.yaml file:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: em-agent
  namespace: default
  labels:
    component: em-agent
spec:
  selector:
    matchLabels:
      component: em-agent
  template:
    metadata:
      labels:
        component: em-agent
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      volumes:
      - name: hugepage
        emptyDir:
          medium: HugePages
      - name: log
        hostPath:
          path: /var/log/nitro_enclaves
      - name: socket-path
        emptyDir: {}
      - name: node-data
        hostPath:
          path: /tmp/em-agent-nitro
      containers:
      - name: em-agent
        image: "fortanix/em-agent-nitro:latest"
        resources:
          limits:
            smarter-devices/nitro_enclaves: "1"
            hugepages-2Mi: 512Mi
            memory: 2Gi
            cpu: 250m
          requests:
            smarter-devices/nitro_enclaves: "1"
        volumeMounts:
        - mountPath: /dev/hugepages
          name: hugepage
          readOnly: false
        - name: log
          mountPath: /var/log/nitro_enclaves
        - name: socket-path
          mountPath: /run/nitro_enclaves
        - name: node-data
          mountPath: /tmp/em-agent-nitro
        ports:
        - containerPort: 9092
          name: http
          protocol: TCP
          hostPort: 9092
        env:
        - name: AGENT_MANAGER_AUTH_BASIC_TOKEN
          valueFrom:
            secretKeyRef:
              name: em-token
              key: token
        - name: MANAGER_ENDPOINT
          value: "ccm.test.fortanix.com"
        - name: MALBORK_LOG_DEBUG
          value: "true"
        - name: NODE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

Run the following command to verify that the Fortanix CCM Nitro Node Agent pod is running:

kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
em-agent-fqp8j   1/1     Running   0          30m

NOTE
Available node agent image tags are listed at https://hub.docker.com/r/fortanix/em-agent-nitro/tags.

8.0 Reinstall the Fortanix CCM Node Agent in a Different Account

Perform the following steps to reinstall the node agent in a different Fortanix Armor account:

Run the following command to delete the /tmp/em-agent-nitro parameter from each worker node:

# log into the node through debug container.
$ kubectl debug node/<node-name> -it --image=busybox
# inside container now run chroot to access the node.
$ chroot /host bash
# Delete the file
$ rm -rf /tmp/em-agent-nitro
# Exit twice to get out of the container
$ exit

Run the following command to create a new em-token secret:

$ kubectl delete secret em-token
$ kubectl create secret generic em-token --from-literal=token=<join-token-from-account>

Run the following command to restart the em-agent DaemonSet:
```
$ kubectl rollout restart ds em-agent
```

9.0 Converting Nitro Enclave OS Application

For more information on how to create an application in Fortanix CCM, refer to Add Application.

9.1 Creating NGINX Deployment

The following is the content of the app-deployment.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-pod
  labels:
    app: my-pod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-pod
  template:
    metadata:
      labels:
        app: my-pod
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - name: http
          containerPort: 80
        imagePullPolicy: Always
        securityContext:
          privileged: true
        resources:
          limits:
            smarter-devices/nitro_enclaves: "1"
            hugepages-1Gi: 2Gi
            memory: 2Gi
            cpu: 250m
          requests:
            smarter-devices/nitro_enclaves: "1"
            hugepages-1Gi: 2Gi
        volumeMounts:
        - mountPath: /dev/hugepages
          name: hugepage
          readOnly: false
        - name: log
          mountPath: /var/log/nitro_enclaves
        - name: socket-path
          mountPath: /run/nitro_enclaves
        env:
        - name: NODE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: NODE_AGENT
          value: http://$(NODE_IP):9092/v1/
        - name: RUST_LOG
          value: debug
      volumes:
      - name: hugepage
        emptyDir:
          medium: HugePages
      - name: log
        hostPath:
          path: /var/log/nitro_enclaves
      - name: socket-path
        emptyDir: {}

Update the container image details to deploy a different application build. Run the following command to deploy the application:

kubectl apply -f app-deployment.yaml

Documentation Index