Azure Kubernetes Service with Fortanix Confidential Computing Manager

December 18, 2020 07:56

USING AZURE KUBERNETES SERVICE WITH CONFIDENTIAL COMPUTING MANAGER USER GUIDE.pdf
(300 KB)

Introduction

This article describes how to set up an Azure Kubernetes Service (AKS) cluster as worker nodes in Fortanix Confidential Computing Manager (CCM).

Setup AKS Cluster as Worker Nodes in Fortanix CCM

Set up an SGX-capable cluster. For instructions refer to the URL:
https://docs.microsoft.com/en-us/azure/confidential-computing/confidential-nodes-aks-get-started
NOTE
Enable SGX quote helper while creating AKS.
```
--enable-sgxquotehelper
```
For example:
```
az aks create -g myResourceGroup --name myAKSCluster --generate-ssh-keys --enable-sgxquotehelper --enable-addon confcom
```
Run kubectl get ds -n kube-system to see the configured DaemonSets in the kube-system namespace.
When you create the AKS cluster with the addon -confcom, the sgx-plugin and sgx-webhook DaemonSet is set up by default. This is the recommended method.

If the sgx-plugin DaemonSet is not present, follow the instructions in Deploying the SGX device plugin to set up Microsoft’s SGX device plugin in your cluster.
```
kubectl create -f https://github.com/Azure/aks-engine/raw/master/docs/topics/sgx/device-plugin.yaml
```

The sgx-quote-helper DaemonSet will be up and running when you enable sgxquotehelper in step 1. This is the recommended method to create the sgx-quote-helper DaemonSet.
If the sgx-quote-helper DaemonSet is not present, use the YAML file below and install the SGX quote helper.

kubectl create -f sgx-quote-helper-1.17.yaml

File contents of sgx-quote-helper-1.17.yaml:

# Modified from the pre-1.17 version supplied by MS over email.
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: sgx-quote-helper
  namespace: kube-system
  labels:
    app: sgx-quote-helper
spec:
  selector:
    matchLabels:
      app: sgx-quote-helper
  template:
    metadata:
      labels:
        app: sgx-quote-helper
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node.kubernetes.io/instance-type
                operator: In
                values:
                - Standard_DC2s
                - Standard_DC4s
                - Standard_DC1s_v2
                - Standard_DC2s_v2
                - Standard_DC4s_v2
                - Standard_DC8_v2
              - key: kubernetes.io/os
                operator: In
                values:
                - linux 
      containers:
      - name: sgx-quote-helper
        image: mcr.microsoft.com/aks/acc/sgx-attestation:0.1
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            sgx.intel.com/epc: "1Mi"
        volumeMounts:
        - name: var-log
          mountPath: /var/log
        - name: var-run-aesmd
          mountPath: /var/run/aesmd
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - ALL
      volumes:
      - name: var-log
        hostPath:
          path: /var/log
      - name: var-run-aesmd
        hostPath:
          path: /var/run/aesmd

Retrieve the join token for your CCM zone from the CCM UI and store a copy as a Kubernetes secret in your cluster.
1. Navigate to the COMPUTE NODES tab in the CCM UI and click “+ ENROLL NODE to bring up the token dialog. Copy the token. Figure 1: Enroll node
2. Create a Kubernetes secret for the token. Replace the <token> value below with your token.
```
kubectl create secret generic em-token --from-literal="token=(token-from-ccm.fortanix.com)" 
```

Save and download the following YAML file.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: em-agent
  namespace: default
  labels:
    component: em-agent
spec:
  selector:
    matchLabels:
      component: em-agent
  template:
    metadata:
      labels:
        component: em-agent
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      volumes:
      - name: em-agent-data
        hostPath:
          path: /mnt/em-agent-data
      - name: dev
        hostPath:
          path: /dev
      - name: var-run-aesmd
        hostPath:
          path: /var/run/aesmd
      - name: agent-manager-auth
        secret:
          secretName: agent-manager-auth
      containers:
      - name: em-agent
        image: "fortanix/em-agent"
        resources:
          limits:
            sgx.intel.com/epc: "10Mi"
          requests:
            sgx.intel.com/epc: "10Mi"
        volumeMounts:
        - name: em-agent-data
          mountPath: /var/opt/fortanix/em-agent/node
        - name: dev
          mountPath: /dev/host
        - name: var-run-aesmd
          mountPath: /var/run/aesmd
        ports:
        - containerPort: 9092
          name: http
          protocol: TCP
          hostPort: 9092
        env:
        - name: AGENT_MANAGER_AUTH_BASIC_TOKEN
          valueFrom:
            secretKeyRef:
              name: em-token
              key: token
        - name: NODE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

Deploy the node agent DaemonSet.
```
kubectl create -f agent-daemonset.yaml
```
Include the following definitions in the k8s specifications of containers that use SGX. Adjust the requested amount of Electronic Product Code (EPC) memory to suit your node type and workload.
```
resources:
  limits:
    sgx.intel.com/epc: "10Mi"
env:
- name: NODE_IP
  valueFrom:
    fieldRef:
      fieldPath: status.hostIP
- name: NODE_AGENT_BASE_URL
  value: http://$(NODE_IP):9092/v1
```
For more information about scheduling pods with EPC reservations, see Scheduling Pods to TEE enabled Hardware.