Fortanix Data Security Manager Architecture

Introduction

Fortanix Data Security Manager™ (DSM) lets users and applications generate, manage, and store keys, and use them for cryptographic operations. Fortanix DSM is built using Intel® Software Guard Extensions (Intel® SGX) technology available in Intel® Xeon® CPUs. The keys and data are available only inside secure Intel® SGX enclaves, where they are protected from; malware, the operating system, hypervisor, and the service provider, such as the cloud infrastructure provider, or Fortanix. This eliminates the need to rely on the characteristics of physical boxes, such as hardware security modules (HSMs), to provide equivalent security. Our Fortanix DSM product provides HSM-grade security with software flexibility and supports the deployment of applications using public cloud infrastructure, on-premise hardware, or a hybrid infrastructure model that combines both options.

Using the capability of Intel® SGX to secure any x86 code; Fortanix DSM runs the application logic such as authentication, role-based access control (RBAC), password reset, and so on, inside Intel® SGX enclaves as well. Consequently, Fortanix DSM provides a level of security far greater than traditional HSMs. The construction of our FIPS 140-2 Level 3 certified hardware appliance from off-the-shelf hardware also reduces the initial as well as operational costs of Fortanix DSM for this mode of deployment.

Fortanix DSM is built to be horizontally scalable, which allows the capacity of the system to increase linearly as additional nodes are added to a Fortanix DSM cluster. Fortanix DSM is also designed to be highly available, with a high level of redundancy and the ability to recover from disasters. The distributed nature of the encrypted storage system using Cassandra that uses Paxos and Raft provides default protection from the loss of a cluster node or corruption of stored data.

This guide describes the architecture and building blocks of Fortanix DSM with an emphasis on security and scalability.

Design Principles

The following design principles of Fortanix DSM will be useful in understanding the architecture of Fortanix DSM.

  • Secure by Design: By design, Fortanix has no visibility into keys, users, policies, or logs of a Fortanix DSM cluster deployed for a customer. Keys and associated metadata are encrypted at all times, whether in memory or when stored on the disk. This is done using keys that are derived from secrets embedded in the fabric of the CPU, and these keys are available only inside an authenticated Intel® SGX enclave. This ensures that all customer data is protected from malware, rootkits, insiders, and any vulnerability in the operating system.
  • API Driven: Fortanix DSM is designed to use REST APIs for all types of operations – system administration, account management, key management, as well as cryptographic operations. This allows for a high degree of automation in the operation as well as the usage of Fortanix DSM.
  • Infinitely Scalable: Fortanix DSM has a built-in load balancer that allows API requests to be distributed across the nodes in the cluster. All nodes in the cluster are equally capable of serving any API request. This allows for Fortanix DSM capacity to increase linearly as the cluster size is increased.
  • Highly Available: Fortanix DSM relies on a distributed database implementing the Raft consensus algorithm to provide strong consistency. This also allows Fortanix DSM to be available as long as a majority of nodes are available in the aggregate cluster. Where a majority of nodes are unavailable, the system will revert to read-only functionality to maintain service and protect the database, even down to a single functioning node.
  • Disaster Recovery built-in: Fortanix DSM can be deployed as a single cluster instance across multiple data centers that can be distributed geographically. This allows for Fortanix DSM to be available even when a complete site is down, providing high system resilience.
  • Shared nothing architecture: Fortanix DSM is designed with a shared-nothing architecture, where every node is independent and self-sufficient. This allows for ease of cluster management, scaling up cluster size by adding more nodes and replacing faulty nodes.

Architecture

Fortanix DSM deployment contains the following components.

L4LoadBalancer.png
Figure 1: Architecture of Fortanix Data Security Manager Cluster

  • Virtual IP and Load Balancing: A virtual IP is assigned to a Fortanix DSM cluster which can be assigned to any of the healthy nodes in the cluster at any time. This is done using the Keepalived service running on all the nodes in the cluster. This floating IP address should be assigned a name that the organization’s DNS server can resolve. All nodes in a Fortanix DSM cluster use the Kubernetes load balancer, which forwards incoming requests to the individual Fortanix DSM cluster nodes in a round-robin.
  • Database: Fortanix DSM uses the Cassandra database which is a distributed database that has an instance running on every node in the cluster. The database uses a consistency protocol that requires a quorum of nodes to be available:  (n/2)+1 for a cluster of size n. This defines a minimum cluster size of n=3 nodes.
  • Kubernetes: The Fortanix DSM application runs inside a container, which is executed and orchestrated using Kubernetes. Part of the initial setup of nodes involves setting up a Kubernetes cluster and making the nodes part of that cluster. Provisioning of Fortanix DSM and software upgrades are done using Kubernetes commands thereafter.
  • Intel SGX: The entire Fortanix DSM software runs inside an Intel® SGX enclave. This requires the Intel® SGX driver to be installed on the host, and the Intel® SGX platform software (PSW) to be installed in the Fortanix DSM container.
  • Fortanix Data Security Manager Container: The Fortanix DSM application runs inside a Docker container. This includes several critical components of the application, such as TLS termination, the webserver which parses REST APIs, the crypto library, key management logic, as well as the database driver.
  • Fortanix Data Security Manager Web UI: The Fortanix DSM UI is served through a Nginx web server running inside the Fortanix DSM container.
  • Monitoring Software: Every Fortanix DSM node runs a monitoring agent (Sensu). A System Administrator can view alerts from the monitoring dashboard.

Figure 2 shows the architecture of a single Fortanix DSM node:

DSM_BlockDiagramNew.png
Figure 2: Architecture of a single Fortanix Data Security Manager Node

Security by Design

Fortanix DSM is built to protect the customer keys and associated data. All data sent and received from customers is transferred over a secure TLS connection, which is processed inside the boundary of an Intel® SGX enclave, and data is stored in permanent storage after being encrypted using a key known only to the Fortanix DSM cluster and available only inside the enclave. Secret data is never exposed unencrypted outside of a secure Intel® SGX enclave.

Key Hierarchy

Hierarchy of Keys Used in Fortanix Data Security Manager FX2200 (SGX)

Figure 3 describes how various keys are derived or generated in Fortanix DSM, and how they are used to secure data at various points within the system. As shown in Figure 3, customer keys are always kept encrypted in Fortanix DSM, whether they are on the disk, on the network, or in the memory of a Fortanix DSM node:

key-hierarchy-horizontal-SGX1.png
Figure 3: Hierarchy of Keys used in Fortanix DSM FX2200 (SGX)

Hierarchy of Keys Used in Fortanix Data Security Manager (non-SGX)

Figure 4 below describes how various keys are derived or generated in Fortanix DSM (non-SGX). This Key Hierarchy applies only to AES keys.

key-hierarchy-horizontal-Non-SGX-New.png
Figure 4: Fortanix Data Security Manager Key Hierarchy (Non-SGX)

For more details, refer to Running Fortanix DSM in a non-SGX virtual environment.

Key Descriptions

  • Device Master Key – This key is CPU-specific. It cannot rotate.
  • DSM Device Master Key (DMK) – The DMK is derived* using information about the system (BIOS Version, DSM Version) as well as the Device Master Key as inputs. This key is derived at the service start time. It is never written to storage. This key is AES 256-bit.
  • DSM Cluster Master Key (CMK) – The CMK is generated** when the Fortanix DSM cluster create command is issued. This is a randomly generated key using a NIST-approved deterministic random bit generator (DRBG). This key is AES 256-bit. The CMK is distributed to the other cluster nodes over TLS and then wrapped with their respective DMK. This key is distributed to other Fortanix DSM nodes over TLS. This key can be rotated as needed by the customer. Rotation of the CMK requires a rolling restart of all nodes in the cluster.
  • DSM Account Key – There is a unique key for each account. This key is AES 256-bit. This key cannot be rotated by itself. It is rotated when the CMK is rotated.
    Triggers for the account key creation/re-creation are as follows:
    • Account creation – At this time, the Account key is generated using CTR DRBG**.
    • Rotation of CMK – At this time, a new account key is generated by doing key derivation from the new CMK using the key derivation function (KDF)*.
    When the CMK is rotated, a new account key is created for each account and stored along with the current account key. The process of new account key creation happens lazily. The new account key is protected using the new CMK. Data protected by an account key is lazily migrated and re-wrapped using the new account key. This re-wrapping happens whenever the data is updated. Any new data created uses the new account key.
    Currently, there is no mechanism to query and find out which objects in an account have completed migration after CMK rotation.
  • Memory Key – The Memory Key is a function of SGX, used to encrypt all data running in memory. It is never written to storage.
  • Kubernetes client certificates – Kubernetes requires PKI certificates for authentication over TLS. These certificates are automatically generated. By default, client certificates generated by kubeadm expire after 1 year. A cron job automatically runs every month. It rotates the private key and renews the certificate. This same renewal script can also be called on-demand to rotate certificates and keys at any time.
  • Etcd certificates – Etcd requires certificates for communication over TLS. These certificates that the Kubernetes cluster requires are automatically generated. Fortanix DSM has a cron job that automatically runs every month. It rotates the private key and renews the certificate.
  • Cassandra TLS Certificates – The TLS key and certificate used by Cassandra containers are ephemeral and last for a short time. It is regenerated and re-signed by Kubernetes Root CA every time the container starts.
  • Cluster TLS Key – The Cluster TLS Key cannot be rotated. The rotation support is in the Fortanix DSM release roadmap.
  • UI Assets Service TLS Key – The UI Assets Service TLS Key cannot be rotated. The rotation support is in the Fortanix DSM release roadmap.
  • Kubernetes Root CA – This is created when Kubernetes is initialized for the first time. Kubeadm does not support rotation or replacement of CA certificates out of the box. Our architecture allows the rotation of all keys and certificates that Kubernetes Root CA signs.
  • Etcd Root CA – This is created when Kubernetes is initialized for the first time. The CA certificate rotation is currently not supported. Our architecture allows the rotation of all keys and certificates that etcd Root CA signs.
NOTE
* Key Derivation - NIST SP 800-108 KDF in Feedback Mode
** Key Generation - NIST SP 800-90A CTR_DRBG

Remote Attestation

Remote attestation is used to establish trust between the nodes in the Fortanix DSM cluster. When a new node joins the cluster, it first needs to provide an attestation report to the cluster where a trusted party attests that the node is running the right Fortanix DSM software on a genuine Intel® SGX processor. The cluster can then verify the attestation, and add the node to the cluster, which results in a cluster master key being provided to the new node.

Role-Based Access Control

Customers need to identify individuals for the following Fortanix DSM roles.

In some cases, the same individual can serve multiple roles. However, Fortanix recommends that the high-value keys inside Fortanix DSM still require a quorum approval process.

  1. System Administrator (SA):
    A user with this role is responsible for installing, configuring, upgrading, and managing Fortanix DSM. Key management and cryptographic operations are not available for SA.
    The following tips can help you identify the right person in your organization for System Administrator (SA) role.
    • SA does not have access to any keys or cryptographic operations, so they need not be familiar with encryption.
    • SA is responsible for procuring Fortanix DSM appliances, deploying them in the data center, upgrading the software, and monitoring for hardware failures.
    • SA should be familiar with assigning IP addresses and other aspects of network management.
    • SA plays the most critical role in ensuring Fortanix DSM clusters are healthy and functional. If all System Administrators were to lose their credentials simultaneously, you will not be able to ever have an SA again on a running cluster. Fortanix cannot help to create a new System Administrator, as having such a capability would compromise your security. Users and applications would still be able to use the service with existing software, but you cannot update Fortanix DSM software without an active SA.
    • Fortanix recommends at least two System Administrators to avoid disruption of the service in the case that a System Administrator leaves the organization or credentials are lost.
    • After the initial deployment, SA is not involved with the daily operations of Fortanix DSM, but their services will be critical and needed when bringing in new appliances to add to the Fortanix DSM cluster.
  2. System Operator (SO):
    A user with this role is responsible for monitoring and troubleshooting Fortanix DSM. Key management and cryptographic operations are not available for SO.
    Think of the System Operator (SO) as an extended team member for the System Administrator. The SO role is useful for large-scale and geographically distributed deployment of Fortanix DSM. The SO role is optional for smaller deployments and can be combined with the SA role.
    • SO is familiar with the organization’s infrastructure monitoring tools including alerts, pager duties, and so on.
    • SO monitors the Fortanix DSM 24x7 for malfunction and takes remedial actions as needed.
    • In a typical case, SO is part of the operations team and FortanixDSM monitoring is just one of their many duties.
  3. Account Administrator (AA):
    A user with this role can create an account, and add groups, users, and applications to the account.
    Account Administrator (AA) typically works for a business unit or a centralized crypto team. In either case, AA is the main champion of encryption for the business unit.
    • AA has a good understanding of encryption and key management. At the minimum, AA knows why Fortanix DSM is needed for the business unit and ensures that those needs are met with proper integration.
    • AA is also the gatekeeper of users and applications in the account. AA adds users to the right groups and gives them various permissions. An Account Administrator can add additional Account Administrators to the account.
    • AA plays the most critical role in ensuring the continuity of Fortanix DSM service. If all Account Administrators were to lose their credentials simultaneously, you will not be able to ever have an AA again on that account. Fortanix cannot help to create a new Account Administrator for the account, as having such a capability would compromise your security. Users and applications would still be able to use the keys without an active AA, but you cannot add new users without an active AA.
    • Fortanix recommends at least two Account Administrators to avoid disruption of the service in the case an Account Administrator leaves the organization.
    • An acceptable approach is to have two prominent individuals in the organization with each being both Account Administrator and System Administrator.
  4. Account Member (AM):
    An Account Member is an administrator or auditor for one or more groups in an account. As a group administrator, an account member may create applications and security objects in the group and can invite other users to the group. As a group auditor, an account member may only view objects and their associated audit logs in the group.
    Account Members are regular users of Fortanix DSM. They can create keys and add applications to perform cryptographic operations. 
    • You can have as many AMs as you like but the defense-in-depth strategy suggests that you have only those team members to your account who have legitimate business reasons. You may also create a demo account for team members to experiment with Fortanix DSM.
  5. Account Auditor (AAU):
    A user with this role can view various entities in the account and their associated audit logs but cannot perform any action.
    Account Auditor needs some familiarity with cryptography to look for signs of abuse in Fortanix DSM. It is also common to have an Account Administrator play the role of Account Auditor in the early days of the service.
    • Account Auditors can use GUI to look at the log, or they can use Splunk or other SIEM to import logs from Fortanix DSM and analyze them there.
  6. Applications (App):
    An application can programmatically access Fortanix DSM and can perform cryptographic operations in addition to key management operations.
    Applications use Fortanix DSM for various cryptographic operations such as encryption, decryption, signing, and so on.
    • Applications can authenticate to Fortanix DSM using an API key or a client TLS certificate.
    • An Account Member or Account Administrator is typically responsible for identifying relevant applications and integrating them with Fortanix DSM.

The Fortanix DSM role-based access control model is shown in Table 1.

Access Control for Keys

Access_Control1.png
Figure 5: Fortanix Data Security Manager applications, groups, and keys

A key (or Security Object) in Fortanix DSM can be managed by a user or an application. This includes actions such as generating a key, importing a key, disabling a key, changing permissions on a key, or deleting a key. A key, however, can be used for cryptographic operations only by an application. A key resides in a group in Fortanix DSM, and it can be accessed only by users and applications that also belong to that group.

A typical use case for Fortanix DSM in a large enterprise may involve providing access to keys to several applications that operate on the same encrypted data set but are owned by separate business units or product owners. Fortanix DSM makes it very easy for applications to either share keys or keep access to their set of keys isolated.

  • Key sharing between applications: Sharing keys between applications is as easy as making the members of the application of the same group. An application in Fortanix DSM may be a member of multiple groups and has a default group that is used by APIs if no group is explicitly defined for a given operation (for example: key import). The application has full access to all the keys in all the groups it belongs to. For example, in Figure 4, Application A has full access to all the keys that Application B generates due to its membership in Group B.
  • Partitioning of keyspace between applications: Fortanix DSM also allows applications to keep their keyspace separated by making them members of a disjoint set of groups. For example, in Figure 4, Application C does not have access to any of the keys available to Applications A and B as it does not share any group with those applications.

The group membership of applications is determined either by the Account Administrator (AA), or the Account Member (AM) if the AM has a group administrator role for the group, he is providing access to. The Fortanix DSM users must ensure how applications are provided access to keys in Fortanix DSM.

Credential Management

Fortanix DSM supports SSO-based authentication using SAML 2.0, as well as password-based authentication for users. For password-based authentication, Fortanix recommends enabling two-factor authentication. 2FA is currently supported using the FIDO2/WebAuthn standard, which is supported by devices such as YubiKey. This also works for FIDO U2F authenticators since FIDO2/WebAuthn is backward compatible with them. Users must manage their passwords securely. If a user loses her password, she can reset her password by getting a password change token emailed to her.

 

For applications, Fortanix DSM supports two forms of authentication. Applications can use either a system-generated API Key or a client TLS certificate to authenticate themselves to Fortanix DSM. The form of authentication is established at the time an application is added to Fortanix DSM by an Account Administrator or Account Member.

FIPS Validation

The Fortanix DSM appliance is certified to FIPS 140-2 Level 3 standard (NIST Certificate 3545), while the software itself is certified, separately, to FIPS 140-2 Level 1 standard (NIST Certificate 3326). The relevant listings on the NIST website can be found here:

The details of the approved cryptographic algorithms that are used by Fortanix DSM in FIPS-approved mode of operation are here.

Best Practices

Fortanix recommends the following best practices for an on-premises customer:

  • An organization may choose to create a single account or multiple accounts. While a single account may be easier to manage and can be centrally administered, multiple accounts may be desirable for an organization with multiple business units that may want their account administrators to correspond to their accounts.
  • For every account, it is recommended to have two Account Administrators and one Account Auditor.
  • In every account, there may be multiple applications that integrate with Fortanix DSM. The Account Administrator may choose to create a group for every application or use case and add the application owner as an account member and the group administrator for that group. The Account Member may then invite other members of their team to this group.

Nothing is more paramount than ensuring the keys are protected when they are created. Whether you create a key or import it, take proper precautions:

  • If it is a highly sensitive key (“Will exposing this key cause public damage to my organization”), make sure it requires a quorum approval to be used.
  • If this key should not be exported outside Fortanix DSM, mark it as such. Note that Fortanix DSM allows you to mark keys as non-exportable even after they have been generated.
  • If this key should be used only for signing or only for encryption, mark it as such.

Comments

Please sign in to leave a comment.

Was this article helpful?
1 out of 1 found this helpful