Fortanix Data Security Manager Installation Guide - On-Prem

Fortanix® and the Fortanix logo are registered trademarks or trade names of Fortanix, Inc.

All other trademarks are the property of their respective owners.

NOTICE: This article was produced by Fortanix, Inc. (Fortanix) and contains information that is proprietary and confidential to Fortanix. The article contains information that may be protected by patents, copyrights, and/or other IP laws. If you are not the intended recipient of this material, destroy this article and inform [email protected] immediately.

1.0 Introduction

This article describes the process of installing and configuring the Fortanix-Data-Security-Manager (DSM). This setup is a two-phase process as follows:

• System Installation and Configuration: In this phase, you install the operating system and configure the network on the servers.

• Fortanix DSM Installation and Configuration: In this phase, you install the Fortanix DSM service and configure the Fortanix DSM cluster.

2.0 Quick Setup

To set up IPMI for FX2200 Series 2, refer to the IPMI Setup Guide.

2.1 List of Required Open Ports

For the list of open ports, refer to the Port Requirements.

3.0 Prerequisites

Before beginning the Fortanix DSM installation process, ensure that the following requirements are met:

1. You have at least three Fortanix FX2200 servers

2. Network configuration (IP address, subnet mask, and gateway) has been assigned for each server.

3. If you are not planning on using an external load balancer, an additional IP address should be allocated for Fortanix DSM. This will serve as the cluster IP address to which Fortanix DSM clients will connect.

4. It is recommended assigning two Fully Qualified Domains (FQDNs) for the Fortanix DSM cluster. If the hostnames are assigned in your domain, for example, your-domain.com, then the preferred hostnames are sdkms.your-domain.com, and sdkms-ui.your-domain.com.

5. You should add DNS entries (A records) for the above two FQDNs, all mapped to the cluster IP address described above, to your DNS server.

6. All the ports mentioned in “List of required open ports” should be open between servers.

7. You should have the ability to issue or generate certificates for certificate requests (CSRs) generated by Fortanix DSM with the subject alternative name (SAN) containing the above-stated hostnames.

8. You should be able to configure NTP on the servers. If the servers are not going to have access to public NTP servers, then they need to be able to connect to an NTP server on your network.

9. You should have received the software installation packages from Fortanix. The usual way of distribution is using a downloadable link from https://support.fortanix.com/.

10. Ensure that none of the nodes in the cluster has the same hostname.

    NOTE

    The hostnames MUST be in lower case.

11. If you are using a multi-node cluster, ensure that you update the hostname. Perform the following steps:

    1. Use the command sudo hostnamectl set-hostname name to update the hostname.

    2. Add the hostname in /etc/hosts.

    3. Verify that the hostname is correctly updated using the command cat /etc/hostname.

12. Ensure IPMI IPs are assigned to the servers if you want to remotely manage the servers. For details of the IPMI setup, refer to the IPMI Guide.

13. Get the default username/password for the FX2200 appliance. These will be distributed by email from Fortanix post shipment of the servers to the relevant contact person in your team.

4.0 System Installation and Configuration

4.1 Remote Administration by IPMI

If your server has IPMI (Intelligent Platform Management Interface) setup, then you can remotely configure the server for the rest of the network configuration. For details of the IPMI setup, refer to the IPMI guide.

Once network configuration is completed, then direct SSH can be used for remote login. Perform the following steps for IPMI remote login:

1. Use IPMI web page accessible at the specified IPMI IP address through any browser. For example, http://192.168.1.25/#login

2. Use IPMI credentials provided by the Datacenter team which performed the IPMI setup. Go to Section Remote Control → Console Redirection. This opens a Java console which provides the terminal view of the server. Now boot the server and login using the system administrator credentials provided by the Fortanix team. You can now follow the rest of the steps.

4.2 Network Configuration

Configure a network interface with an IP address, subnet mask, and gateway, such that the servers are reachable from any intended client. You can do this using the console over IPMI if IPMI has been setup.

If you are using Fortanix supplied servers, note that these servers have two network interfaces. You can just use one network interface. If your network topology/deployment requires separating interfaces for external traffic and intracluster traffic, then you may set up both the network interfaces.

For setting up the network, edit the /etc/network/interfaces file to specify IP address, gateway, netmask, and DNS server information. Refer to the following sample file to help you get started.

Example /etc/network/interfaces file:

# This file describes the network interfaces available on your system 
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
  
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto interface_name
iface interface_name inet static
 address xxx.xxx.xxx.xxx
 gateway xxx.xxx.xxx.xxx
 netmask xxx.xxx.xxx.xxx
 dns-nameserver xxx.xxx.xxx.xxx
 dns-nameserver xxx.xxx.xxx.xxx

To set the hostname on each node, in the following file, remove sdkms-server, and replace with the intended hostname.

sudo vi /etc/hostname

In the following file, add the IP hostname and/or FQDN.

sudo vi /etc/hosts

Reboot the system for changes to take effect.

4.3 LACP Bonding Configuration

The Fortanix DSM appliance can be configured to have LACP bonding.

Following is an example of network configuration that shows how to set it up:

1. Run the following command to install “ifenslave” package:

   sudo apt-get install ifenslave

2. Example interfaces file location is /etc/network/interfaces:

   # The ordering of these interfaces is important, see comment for bond0 below.
         
   auto enp80s0f0
   iface enp80s0f0 inet manual
       bond-master bond0
       # This command is important, see comment for bond0 below.
       pre-up ifup bond0 &
   
   # The ordering of these interfaces is important, see comment for bond0 below.
   auto enp80s0f1
   iface enp80s0f1 inet manual
       bond-master bond0
       # This command is important, see comment for bond0 below.
       pre-up ifup bond0 &
   
   # The ifenslave scripts are written for all interfaces being brought up at once. However, `ifup` will bring interfaces up one-by-one. Without doing anything extra, this will result in a deadlock or a 60-timeout, depending on the order of the master and slave interfaces. We make sure to bring up the bond master simultaneously with a slave, by running the ifup command for the master in the background using `pre-up`. `ifup` implements locking and state checking to make sure that a single device is only brought up once.
   auto bond0
   iface bond0 inet static
       bond-miimon 100
       bond-mode 802.3ad
       bond-slaves yes
       address ...
       gateway ...
       netmask ...
       dns-nameservers ...

5.0 Fortanix DSM Installation and Configuration

This section describes the tasks to install the Fortanix DSM software and configure the Fortanix DSM cluster.

5.1 Install Fortanix DSM

Perform the following steps to install the Fortanix DSM package on all the servers:

1. The latest installation file is available at Fortanix-Data-Security-Manager-Releases. This needs a customer account, which Fortanix provides for the relevant contact person.

2. Download and copy the Fortanix DSM installation file sdkms_<version>_install.sh to each server. The latest installation files are hosted on https://support.fortanix.com/

3. Run the following command to install the package, replacing the package name with the filename of the package in your Fortanix DSM distribution.

   sudo chmod +x sdkms_2.3.688-124_install.sh
   sudo ./sdkms_2.3.688-124_install.sh

4. Run the following command to reboot the system if a new version of kernel was installed:

   sudo reboot

5. Run the following command to reset on all servers:

   sdkms-cluster reset --delete-data --reset-iptables

5.2 NTP Configuration

NTP is required to be able to run Fortanix DSM. If the servers are not going to have access to public NTP servers, you will need to update the NTP configuration file (/etc/ntp.conf) to specify NTP server(s) on your network. To enable local NTP when an external NTP server is not available, add the directive below to the config.yaml file as referenced in Section 5.4: Configure Other Nodes for Joining the Cluster.

5.2.1 External NTP Options

1. Comment out servers for the NTP Pool Project and NTP server as a fallback section and add the following server IP addresses:

   # pool 0.ubuntu.pool.ntp.org iburst
   # pool 1.ubuntu.pool.ntp.org iburst
   # pool 2.ubuntu.pool.ntp.org iburst
   # pool 3.ubuntu.pool.ntp.org iburst
   
   # Use Ubuntu's ntp server as a fallback.
   # pool ntp.ubuntu.com
   
   server 0.0.0.0

2. Run the following command to restart NTP:

   sudo systemctl restart ntp
   ntpq -p

5.2.2 Internal NTP Options

You can run a local NTP server when an external NTP server is not available. This will run the NTP pod on each node and will keep the time synchronized within the cluster to one of the cluster nodes.

global:
  localntp: true

5.3 Begin Fortanix DSM Configuration

This step needs to be run on only one of the servers.

5.3.1 Setup Deployment Specific Configuration File

Run the following command to copy the template file in the home directory and then edit it:

cp ~/config.yaml.example config.yaml

When deploying Fortanix DSM you have options with respect to the load balancer, remote attestation, internet subnet, and remote syslog logging. Based on the option you select, edit the config.yaml file as explained:

Load Balancer Options:

• Use the built-in load balancer – this is the default option.

  • Edit the config.yaml file to set the correct cluster IP address by adding the value of the parameter “clusterIp” under “sdkms” section.

    sdkms:
      clusterIp: 10.0.0.1
    keepalived:
      nwIface: enp80s0f0

• Use an external load balancer

  • Edit the config.yaml file to add the following parameters under “global” section (consider the yaml  file format and spacing).

    sdkms:
      clusterIp: 1.2.3.4 # mandatory field, set to dummy for external LB
    global:
      externalLoadBalancer: true
    # keepalived:
    # nwIface: enp80s0f0

Remote Attestation Options:

Remote attestation is performed when changes are performed on a Fortanix DSM cluster. Intel SGX using Data Center Attestation Primitives (DCAP) is the supported attestation mechanism for platform attestation. Remote attestation is performed in the following scenarios:

• Joining new nodes to an existing cluster.

• Replacing nodes in an existing cluster (for example: replacing failed hardware nodes).

• Making hardware changes and performing firmware updates.

• Upgrading the cluster software version.

WARNING

Intel has announced the End of Life (EOL) of their Intel Attestation Service (IAS) for Intel SGX (https://community.intel.com/t5/Intel-Software-Guard-Extensions/IAS-End-of-Life-Announcement/td-p/1545831). If you are currently using IAS attestation, you must migrate to DCAP attestation to verify the integrity of the nodes’ CPUs as they are enrolled in a DSM cluster. For detailed migration steps, refer to Section 5.3.2: Migrating Cluster from IAS to DCAP Attestation. After Migration, use any of the following DCAP Attestation configurations as per your requirement.

• Remote attestation enabled.

  • This is the default option and no additional changes to the config.yaml file is required.

  • This option requires that the appliances have internet access so they can connect to the Fortanix attestation proxy service.

• Required URLs for remote attestation.

  • In this scenario, Fortanix DSM recommends setting up an http proxy to the internet. Make sure the following URLs are accessible through the proxy:

    • http://ps.sgx.trustedservices.intel.com:80, port 80

    • https://trustedservices.intel.com/content/CRL, port 443

    • http://trustedservices.intel.com/ocsp, port 80

    • http://whitelist.trustedservices.intel.com/SGX/LCWL/Linux/sgx_white_list_cert.bin, port 80

    • https://iasproxy.fortanix.com, port 443

    • https://pccs.fortanix.com, port 443

    NOTE

    FortanixPCCS will be available from 17th February, 2025.

  • For DCAP online attestation with the Secure Node Join feature enabled, edit the config.yaml file to add the following parameters under the “global” section (consider the yaml file format and spacing).

    global:
      attestation:
        dcap: onlineWithTrustedNodeIdentity

  • For DCAP offline attestation with the Secure Node Join feature enabled, edit the config.yaml file to add the following parameters under the “global” section (consider the yaml file format and spacing).

    global:
      attestation:
        dcap: offlineWithTrustedNodeIdentity

  • For DCAP online attestation using Fortanix Provisioning Certification Caching Service (PCCS), edit the config.yaml file to add the following parameters under the “sdkms” section (consider the yaml file format and spacing).

    NOTE

    For platform attestation, when using pcs: FortanixPCCS, you must whitelist the URL: https://pccs.fortanix.com to create an end-to-end encrypted and authenticated connection.

    global:
      attestation:
        dcap:
          type: online
    sdkms:
      dcapProviderConfig:
        pcs: FortanixPCCS
        fortanix_pccs:
          api_version: 4

  • For DCAP online attestation using Fortanix PCCS using a proxy server to proxy outbound attestation network request instead of whitelisting https://pccs.fortanix.com, use the proxy field in the attestatin.dcap section. Edit the config.yaml file to add the following parameters under the “attestation” section (consider the yaml file format and spacing).

    global:
      attestation:
        dcap:
          type: online
          proxy: {url to http/https proxy server}
    sdkms:
      dcapProviderConfig:
        pcs: FortanixPCCS
        fortanix_pccs:
          api_version: 4

    WARNING

    IAS with DCAP Hybrid Configurations – The following hybrid configurations will work only until IAS EOL (April 2nd, 2025).

  • For IAS and DCAP online attestation using Fortanix PCCS, edit the config.yaml file to add the following parameters under the “global” and “sdkms” sections (consider the yaml file format and spacing).

    global:
      attestation:
        ias:
          url: ftx-proxy
        dcap:
          type: online
    sdkms:
      dcapProviderConfig:
        pcs: FortanixPCCS
        fortanix_pccs:
          api_version: 4

    The proxy field is identical to global proxy, so if you did not set up proxy, please remove it.

  • For IAS and DCAP online attestation using a proxy server to only proxy outbound attestation network request instead of whitelisting https://pccs.fortanix.com, edit the config.yaml file to add the following parameters under the “attestation”  section (consider the yaml file format and spacing).

    global:
      attestation:
        ias:
          url: ftx-proxy
          proxy: url: {url to http/https proxy server}
        dcap:
          type: online
          proxy: url: {url to http/https proxy server}
    sdkms:
      dcapProviderConfig:
        pcs: FortanixPCCS
        fortanix_pccs:
          api_version: 4

• Remote attestation disabled.

  • Edit the config.yaml file to add the following parameters under the “global” section (consider the yaml file format and spacing).

    global:
      attestation: null

    NOTE

    Null attestation to SGX attestation is not supported.

Internal Subnet Options:

• Kubernetes requires two internal networks for intra-cluster communication. By default, use 10.244.0.0/16 and 10.245.0.0/16 when unspecified.

• To configure custom subnet for Kubernetes internal network, set serviceSubnet and podSubnet  under the “global” section (consider the yaml file format and spacing) as shown:

  global:
    serviceSubnet: 172.20.0.0/16
    podSubnet: 172.21.0.0/16

Remote Syslog Logging Setting Options:

Fluentd supports forwarding logs to single syslog server and to multiple syslog servers.

• To configure forwarding the 'container', 'sdkms', and 'system' logs from all the nodes in the cluster to one or more remote syslog servers, set the host, port, protocol, and log_type parameters for each type of log under "advancedSyslog" option of fluentd section of the config.yaml file (consider the yaml file format and spacing).

  NOTE

  The “remoteSyslog” option, which allowed forwarding all logs to one remote Syslog server, will be deprecated in future releases of Fortanix DSM. Use "advancedSyslog" option to use the enhanced log forwarding functionality.

  The valid values for,

  • host: Any valid IP address

  • port: Any valid port

  • protocol: Either tcp or udp

  • log_type: Either one of "CONTAINER", "SYSTEM", "SDKMS"

  The following is an example for advancedSyslog if the forwarding is for 3 syslog servers:

  fluentd:
     advancedSyslog:
       - host:
         port:
         protocol:
         log_type: "CONTAINER"
       - host:
         port:
         protocol:
         log_type: "SYSTEM"
       - host:
         port:
         protocol:
         log_type: "SDKMS"

Log Filtering Options:

There are three kinds of logs produced for every request:

• A connection log, indicating that a new connection has been made and the IP from where the connection is made.

• A request log, indicating that a request has been received, along with the IP from where that request is coming from and the associated HTTP Method and URL.

• A response log, indicating that a response has been returned, along with the associated status code.

When Log Filtering is enabled, it will cache request logs matching a shortlist of method-URL pairs. If the corresponding response is a success, the request and response logs are discarded. If the response is not a success, then both the request and response logs are printed.

The filtered combinations are:

• GET /sys/v1/health

• HEAD /sys/v1/health

• HEAD /

Since Kubernetes and most load balancers create a new connection for each request they make, this feature will attempt to profile individual IPs. If an IP is found to only make health check-related requests (that is, one of the three filter combinations listed above), then the connection logs from these IPs will also be filtered. If one of these IPs later makes a non-filtered request, its connections logs will no longer be filtered until it returns to making only filtered requests.

Log Filtering is enabled by updating the cluster-wide configuration. When enabled, the backend will no longer emit logs associated with successful health checks.

Depending on the cluster, this can amount to a substantial reduction in the volume of logs later forwarded over Syslog. The default is false.

filterConfig:
  useLogFilter: true | false

Hybrid Cluster Configuration:

With Release 3.25, Hybrid clustering of "Series- I and Series II ", "Azure and Series- I or Series- II ", and "AWS and Series- I or Series- II running Fortanix DSM in same (SGX/non-SGX) software mode" is possible. This allows nodes of different types to participate in a single cluster. When using an internal DSM load balancer (that uses keepalived) with hybrid clusters consisting of Fortanix Series-I and Fortanix Series-II appliances, fetch the network interface names (Series-I = eno1 and Series-II = enp80s0f0f0) of the appliances and configure the config.yaml file as follows:

sdkms:
  clusterIp: 10.0.0.1
keepalived:
  nwIface: enp80s0f0,eno1

5.3.2 Migrating Cluster from IAS to DCAP Attestation

You must migrate your Fortanix DSM Series 2 (S2) on-premises cluster from IAS-only to DCAP attestation as IAS attestation is nearing EOL. This migration is required for SGX attestation in your Fortanix DSM cluster.

NOTE

Fortanix DSM Series 1 (S1) does not support DCAP attestation, so you must first migrate your Fortanix DSM S1 appliance to S2 to use DCAP attestation. To migrate to S2, contact Fortanix Support.

Perform the following steps to migrate your existing Fortanix DSM S2 cluster from IAS to DCAP attestation:

1. Update your existing Fortanix DSM cluster to version 4.36 from DSM System Administration user interface (UI).

   NOTE

   If your Fortanix DSM cluster version is lower than 4.36 and you cannot upgrade to 4.36 but still need to migrate to DCAP, you must wait for DSM 4.34 patch 1 or DSM 4.31 patch 3 releases.

2. The existing Fortanix DSM Series 2 cluster should have the following configuration with IAS attestation.  

   NOTE

   Please reach out to Fortanix Support for assistance with modifying the configuration.

   global:
     attestation:
       ias:
         url: fortanix-proxy

3. Run the following command as a system administrator to update the cluster configuration in DSM and the Kubernetes configuration with DCAP attestation to enable the S2 cluster node join:

   sudo sdkms-cluster deploy --stage DEPLOY --config config.yaml

   The following is an example config.yaml file which must be used for DCAP attestation.

   global:
     attestation:
       dcap:
         type: online
   sdkms:
     dcapProviderConfig:
       pcs: FortanixPCCS
       fortanix_pccs:
         api_version: 4

   NOTE

   • FortanixPCCS will be available from 17th February, 2025.

   If you originally have a proxy for IAS, you may have the following configuration:

   global:
     attestation:
       ias:
         url: ftx-proxy
         proxy: {url to http/https proxy server}

   To migrate to DCAP, edit the config.yaml file to copy the proxy field to the attestation.dcap section (consider the yaml file format and spacing) as shown below

   global:
     attestation:
       dcap:
         type: online
         proxy: {url to http/https proxy server}
   sdkms:
     dcapProviderConfig:
       pcs: FortanixPCCS
       fortanix_pccs:
         api_version: 4

4. The above command starts the rolling update of the cluster. Wait for the update to finish. Ensure that all sdkms and aesmd pods are healthy after rolling update completes.

   The cluster will now use DCAP attestation whenever attestation is required.

   NOTE

   If you are already using DCAP online attestation with Intel API version 3, before starting the DSM upgrade, run the following command as a system administrator to fetch the DCAP artifacts. This is not for Azure or IAS attested S2 clusters.

   sudo sdkms-cluster secret set secret-dcap-provider-credentials

   Next, you will be asked to enter your Intel API key for Intel Trusted Services for DCAP provider:

   Please insert your Intel API key for Intel Trusted Services for DCAP provider (use Ctrl+D to finish):

   After you input the Intel API key, press Ctrl+D two times (DO NOT press Enter after providing the API key).

   xxxxxxxxxxxxxxxxxxxxsecret/secret-dcap-provider-credentials configured

   Continue with Steps 1 and 3 above to migrate to DCAP attestation with Fortanix PCCS.

References:

• For more information on using global proxy configuration to use a proxy server to proxy outbound network request, refer to Section 5.3.3: Proxy Support for Outbound Connectors.

• For more information on using DCAP online/offline attestation with Secure Node Join feature, Fortanix PCCS, Fortanix PCCS (proxy), Intel V4, and IAS with DCAP Hybrid attestation, refer to the “Remote Attestation Options” in Section 5.3.1: Setup Deployment Specific Configuration File above.

5.3.3 Proxy Support for Outbound Connections

You can add cluster-wide proxy configuration for outbound connections. This is defined in the yaml configuration files.

The global proxy functionality will not override existing proxy settings in the .global. attestation section.

However, if no proxy settings exist in the .global. attestation section, remote attestation proxy setting will inherit the global settings.

Since the attestation proxy settings does not have an exclude option, the global exclude setting will be added to attestation proxy settings.

The Global proxy functionality is only available in SGX based deployments (FX2200 and Azure Confidential Computing virtual machines). The configuration is defined in the .global.proxy entry. It holds two values:

• url: proxy value (mandatory).

• exclude: Comma separated list of hostnames and optionally port that should not use the proxy (optional).

global:
   proxy:
      url: <proxy_url>
      exclude: <host>,<host:port>

exclude Options:

• The hosts in exclude will be suffix matched. For example: exclude: example.com will match example.com and subdomain.example.com but will not match example.com.net.

• Any leading '.' is removed. For example: exclude: .example.com is the same as exclude: example.com.

• CIDR notation is accepted. For example: 111.222.0.0/16 will match all IP addresses in the format 111.222.xxx.xxx.

• The wildcards (*) or regex are NOT supported, and hostnames are not resolved to IP addresses.

• The exclude entry will automatically contain hostnames used internally.

5.3.4 HAProxy Support

To enable HAProxy support, you must modify the proxyHeaderConfig field of the config.yaml file for the cluster and reapply it with the cluster tool. The field has three possible values:

• disabled is the default condition and is equivalent to the field missing from the config.yaml file.

• "maybefrom=<cidr1>,<cidr2>,...", where the CIDR ranges point to the server(s) initiating the proxy protocol enabled connection.

• requiredfrom=<cidr1>,<cidr2>,...", where the CIDR ranges point to the server(s) initiating the proxy protocol enabled connection.

The minimum format is:

global:
   proxyHeaderConfig: disabled
   # OR
   proxyHeaderConfig: "requiredfrom=<cidr>,<cidr>,..."
   # OR
   proxyHeaderConfig: "maybefrom=<cidr>,<cidr>,..."

There are two options to securely handle transitions in accordance with the PROXY header spec. The sequence for enabling support is:

1. The proxyHeaderConfig field is disabled.

2. If you have an extant cluster that does not use proxy protocol, the process for enabling it is to first apply the maybefrom variant to the pods, with the given CIRDs pointing to the IPs of the load balancer. These should be as specific as possible. Once that is done, the proxy header is enabled on the load balancer serving the cluster.

3. Next, the config is changed to the requiredfrom variant with the same CIDRs. On fresh clusters, the header can start off as requiredfrom, if the load balancer already has proxy support enabled.

To disable it, the opposite steps are followed, that is:

1. First, the config is changed to maybefrom.

2. The load balancer's proxy support is disabled, and then the config is changed to disabled.

5.3.5 Sealing Key Policy

A Sealing key is used to wrap a cluster master key. Each node has its own unique Sealing key. Sealing key is derived and not saved anywhere in the system.

The Sealing Key policy defines what type of Sealing key should be used in the cluster.

Possible values are:

• SGX - This is the default value. With this policy SGX sealing key is used. SGX sealing key cannot be retrieved from outside SGX. This provides security guarantees for sealing key.

• Personalization Key - Personalization key is generated using Fortanix DSM Deterministic random bit generator (DRBG) and stored in tamper protected key storage module. In this policy type, the sealing is derived using SGX sealing key and personalization key. This provides additional protection of sealing key. Personalization key is zeroized upon tamper, which in turn disallows deriving the sealing key.

• Recovery Key – In this policy type, the sealing key is derived using personalization key and a recovery key. Recovery key is automatically generated and can be retrieved by a sysadmin. In this policy, in addition to using personalization key based sealing key for wrapping cluster master key, recovery key is also used to wrap the cluster master key. This allows the sysadmin to recover a node using recovery key in case personalization key is zeroized. After the setup, a sysadmin must retrieve the recovery key and store securely.

Setup

When setting up a fresh cluster, add the following lines under the “global” section in config.yaml and provide one of the values mentioned above:

sealingKeyPolicy: VALUE

Example:

sealingKeyPolicy: personalizationKey

Node Recovery

If the personalization key or recovery key policy is used and if the personalization key is zeroized, then the node goes in "NeedsReocvery" state. In this state for the Fortanix DSM pod on this node to become functional, recovery needs to be performed. Recovery will allow the node to unwrap the corresponding cluster master key blob.

To perform recovery, run following script as root:

/opt/fortanix/sdkms/bin/node_recover.sh

Recovery can be done with or without a recovery key.

• If the recovery key is not available or if the sealing key policy value is "personalizationKey", then recovery requires that there be other working nodes in the cluster. In that case run the command as follows:
  

  sudo /opt/fortanix/sdkms/bin/node_recover.sh --node-name NODE_HOST_NAME

  Where,

  • NODE_HOST_NAME is node name where recovery is to be performed.

• If the recovery key is available and the sealing key policy is "recoveryKey", then recovery can be done even on a standalone node. In that case run the command as follows:
  

  sudo /opt/fortanix/sdkms/bin/node_recover.sh --node-name NODE_HOST_NAME --recovery-key
      RECOVERY_KEY_VALUE

  Where,

  • NODE_HOST_NAME is node name where recovery is to be performed.

  • RECOVERY_KEY_VALUE is the value of recovery key.

Updating sealing key policy in the existing cluster

For existing clusters, the sealing key policy is by default "sgx". The procedure for changing the sealing key policy is as follows:

1. Update config.yaml to add one of following lines under the “global” section:

   sealingKeyPolicy: personalizationKey
   sealingKeyPolicy: recoveryKey

2. Apply the updated config by running following command:

   sudo sdkms-cluster deploy --config ./config.yaml --stage DEPLOY

3. Check if the new value was applied by running the following command and check the value of "sealingKeyPolicy".

   sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl get cm config-values -oyaml

4. Run the following command:

   sudo /opt/fortanix/sdkms/bin/update_sealing_key_policy.sh

5. The policy change will be effective on the next upgrade.

5.3.6 Create Fortanix DSM Cluster

1. Run the following command to create a Fortanix DSM cluster:

   sudo sdkms-cluster create --self=<server ip address/subnet mask> --config ./config.yaml

2. Check the status of all pods to verify all pods are running correctly. Run the following command to list the pods and their status:

   sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl get pods -o wide

Verify the Node Status in the Fortanix DSM System Administration View

If you have enabled the Secure Node Join feature with dcap:onlineWithTrustedNodeIdentity or dcap:offlineWithTrustedNodeIdentity attestation in the Remote Attestation Options as described in Section 5.3.1: Setup Deployment Specific Configuration, then as a system administrator, you can log into the Fortanix DSM System Administration view.

1. Navigate to System Administration → CLUSTER → TRUST CENTER tab.

2. The first node and its node identity will be visible in the TRUST CENTER table.

   

The Fortanix DSM related pods may continue to crash until the certificates are installed Section 5.5: Install Certificates, but verify that the pods related to aesmd, Elasticsearch, and Cassandra are running.

If you want to set up a cluster that spans multiple sites or data centers, refer to the steps in the Data Center Labeling guide to label which nodes are in what data center.

5.3.7 Configure Data Center Labeling

After all nodes have successfully joined the cluster, you must perform data center labeling. Data center labeling configuration in a multi-site deployment improves read resiliency by enabling requests to read data from the local data center and supports the Read-Only mode of operation when a global quorum is lost and a local quorum is available. Refer to the Fortanix DSM Data Center Labeling guide (use the automated script to configure DC labeling). Refer to the Fortanix DSM Data Center Labeling guide (use the automated script to configure DC labeling).

For more details, refer to Fortanix DSM Read-Only Mode of Operation.

5.4 Configure Other Nodes for Joining the Cluster

Now that the installation is complete, join all other nodes to the cluster by running the following join command:

• If your nodes are in the same subnet, run the following command:

  sudo sdkms-cluster join --peer=<server ip address/subnet> --token=<kubeadm-token>

• If your nodes are in the different subnets, run the following command:

  sudo sdkms-cluster join --peer=<server ip address> --token=<kubeadm-token> --self=<node IP address>

In the above specification,

• Server IP Address corresponds to the node’s IP address on which cluster was created. Specify it with the subnet.

• The node IP address corresponds to the IP address of the joining node.

• If this peer address belongs to a cluster enabled with the Secure Node Join feature with dcap:onlineWithTrustedNodeIdentity or dcap:offlineWithTrustedNodeIdentity attestation, the node will not join the cluster immediately. Instead, the system administrator must log into the Fortanix DSM UI, navigate to System Administrator → CLUSTER → TRUST CENTER tab, and decide whether to approve or decline the node's identity request.

  

  NOTE

  When a new node joins an existing cluster with secured node join enabled, you can find the new node’s NODE IDENTITY in any of the sdkms pod logs in the current cluster.

  For example, in the sdkms pod log of an existing cluster node, you can see the following log:
  2024-10-21 13:11:33.505Z INFO backend::api::cluster - new node_identity added: ABcajcYepnCqhMnz7yYXndghKFeESvqqujecavX2FVg=

  This same node_identity‘s join request is also visible on the Fortanix DSM UI.

  

  The node_identity is derived from the node's hardware-level identifier known as the Platform Provisioning ID (PPID). The PPID value for an Intel-attested node is extracted from its DCAP attestation report. A SHA256 hash of this value is then base64 encoded to produce the final node_identity.

  1. If the system administrator approves the node join request, the TRUST STATUS value for that node changes to “Trusted”, and if you decline the request the TRUST STATUS value for that node changes to “Rejected”.

  2. The sdkms-cluster join peer commands will only execute successfully if the system administrator approves the node identity. Otherwise, they will fail.

     

     NOTE

     Fortanix DSM system operators can view the node identity requests in the TRUST CENTER tab, but do not have the authorization to accept or decline them.

     

Example join commands for a joining node (10.198.0.65) to a created cluster on a node(10.198.0.66) on a /24 subnet.

sudo sdkms-cluster join --peer=10.198.0.66/24 --token=525073.715ecf923e4ae1db

OR

sudo sdkms-cluster join --peer=10.198.0.66 --token=525073.715ecf923e4ae1db --self=10.198.0.65

The token can also be retrieved by executing on the first host:

sudo kubeadm token list

5.5 Install Certificates

Fortanix DSM requires two SSL certificates for the services, one for the Main API service and another for Static asset service. The Certificate Signing Request (CSR) generation is supported for both these certificates which can be signed by your preferred CA provider.

1. Generate a Certificate Signing Request (CSR) to get an SSL certificate. The CSR contains information (for example common name, organization, country) which the Certificate Authority (CA) will use to create your certificate. Generate CSRs by running the following command on the node where the cluster was created:

   sudo get_csrs

   This script is going to generate Certificate Signing requests for both the Fortanix DSM cluster and UI. It also created the first system administrator(sysadmin) account that can log into Fortanix DSM. Domain names can be provided in two formats.

   1. Simple, where Domain name and SANS can be provided. For example, www.fortanix.com.

   2. Distinguished, where full DN string can be provided. For example, CN=www.fortanix.com O=Fortanix L=Mountain View ST=California.

   There is also an option to add Subject Alternative Names. More than one can be added, each one on a new line.

   The following will be then asked to enter through the script:

   1. Domain name for the cluster (Main)

   2. Domain name for the UI (Assets)

   3. Cluster name

   4. Sysadmin email address

   5. Sysadmin password

   NOTE

   • For authentication using mutual Transport Layer Security (TLS), the standard API key-based authentication interface will not work. To support mutual TLS authentication in Fortanix DSM, add an additional interface as SAN Name (apps.sdkms.dev.com) on the main certificate. For more details, refer to the Mutual TLS Guide.

   • When you create or rotate the CSR, the default TLS key type is RSA2048. You can also use higher TLS key sizes or bits. The supported key types are RSA2048, RSA3072, RSA4096, NISTP256, NISTP384, NISTP521. Refer to DSM 4.34 release notes for a known issue “UI doesn’t load after installing NISTP521 certificates”.

   The CSRs are printed on the terminal named “Cluster Certificate Request” and “UI Certificate Request”. These certificates need to be signed by a Certificate Authority.

   

2. A certificate request is created for both the CSRs, one for Main and one for Assets. You will need to sign both the CSRs from one of the CAs or locally sign it yourself.      

   NOTE

   If get_csrs fails then view the logs at the location /var/log/get_csrs.log.

   To generate a new CSR you can use the optional argument --rotate as below. This will rotate the main and asset certificate TLS key.

   sudo get_csrs --rotate

   This command also prompts for domain names, and at the end, a couple of new CSRs are generated, each for a cluster and UI.

   

   NOTE

   In case the above command does not create certificate signing request (CSR) for “UI Certificate Request”, please contact Fortanix Support.

3. After those newly generated CSRs are signed, run the following command to install those signed certificates:

   sudo install_certs --rotate

   The above command installs the new certificates and removes the old certificates.

   NOTE

   If get_csrs fails then refer to the logs at the location /var/log/get_csrs.log.

4. Run the following command to install the signed certificates once you have them from CA:

   sudo install_certs

   This script asks the user to paste the signed certificate chain (leaf certificate first) for both Main (Cluster) and Assets (UI) domain.

   

   

6.0 Using Fortanix DSM

You can access the Fortanix DSM web interface using the hostname assigned to the Fortanix DSM cluster (for example https://sdkms.<your-domain.com>). For detailed information on Fortanix DSM usage, refer to the Fortanix-Data-Security-Manager-DSM.

7.0 Update the Existing DSM Cluster with Trusted Nodes

Perform the following steps to enable Secure Node Join feature for existing clusters with dcap:online or dcap:offline attestation enabled:

NOTE

Ensure that you have upgraded to Fortanix DSM version 4.31 or higher to enable the Secure Node Join feature.

1. Run the following command to fetch the cluster configuration join policy:

   curl --location --request GET 'https://<ui dns>/admin/v1/cluster' --header 'Authorization: Bearer <bearer token>' --header 'Content-Type: application/json'

   NOTE

   The output of this command must be updated to add Secure Node Join in join-policy as shown in the payload of Step 2.

2. Run the following command to enable the Secure Node Join feature for the existing cluster:

   curl --location --request PATCH 'https://hostname/admin/v1/cluster/join_policy' --header 'Authorization: Bearer <token>' --header 'Content-Type: application/json' --data-raw '{
       "new-policy": {
           "join-policy": {
               "all": ["dcap", "node-ca", "trusted-node-identity"]
           },
           "allowed-sgx-types": {
               "any": ["standard","scalable"]
           }
       }
   }

   

3. Run the following command to navigate to the /bin directory and execute the script for backend rolling cluster:

   cd /opt/fortanix/sdkms/bin
   sudo ./dsm_backend_rolling_restart.sh

   This command will restart the cluster to apply the updated configurations and activate the Secure Node Join feature.

   

4. Run the following command to view the list of all the pod IPs:

   sudo -E kubectl get pods -lapp=sdkms -owide

5. Run the following command from the first node in the cluster to add the node identity for each node:

   curl -kv https://<pod-IP>:4444/admin/v1/cluster/node_identities/add_self -X POST

   

8.0 Add Node to an Existing Fortanix DSM Cluster

1. Run the following command to create the kubeadm token from one of the existing nodes in the cluster:

   sudo kubeadm token create

2. Join the new node(s) with sdkms-cluster join command. Ensure the installer script has been executed on these nodes.

   • If your nodes are in the same subnet, run the following command:

     sudo sdkms-cluster join --peer=<existing node IP>/<subnet-mask> --token=<token>

   • If your nodes are in the different subnet, run the following command:

     sudo sdkms-cluster join --peer=<server ip address> --token=<kubeadm-token> --self=<node IP address>

NOTE

If the Fortanix DSM cluster is enabled with the Secure Node Join feature, then perform the steps to add the trusted nodes as mentioned in Section 5.4: Configure Other Nodes for Joining the Cluster to add a node to the cluster.

9.0 Remove Node from an Existing Fortanix DSM Cluster

1. Run the following command to identify the name of the node to be removed under the header "NAME":

   sudo -E kubectl get nodes

2. Run the following command to remove the node from the cluster using the <node name>:

   sudo sdkms-cluster remove --node <node name> --force

3. Run the following command to reset the node:

   sdkms-cluster reset --delete-data --reset-iptables

4. Run the following command to delete the identity of the removed node from the cluster:

   NOTE

   This step is applicable only if the node is a trusted node. This step ensures that an approval request is created for the system administrator to approve or reject the node upon its rejoining.

   curl --location --request DELETE 'https://<ui dns>/admin/v1/cluster/node_identities/<node identity>' --header 'Authorization: Bearer <bearer token>' --header 'Content-Type: application/json' 

5. Rotate the Cluster Master Key (CMK) of the existing cluster. For more information, refer to the Cluster Mater Key Rotation documentation.

   NOTE

   It is recommended to perform the CMK rotation steps whenever a node is removed from the Fortanix DSM cluster.

10.0 Best Practices

• Only issue accounts to trusted administrators.

• Utilize strong passwords.

• Monitor logs.

• Restrict physical access to the appliance to trusted administrators.

• Create two System Administrator accounts.

11.0 Troubleshooting and Support