Using Fortanix Data Security Manager to Secure Servers and Client Keys

1.0 Introduction

This article provides detailed steps for securing Secure Shell (SSH) workflows using Fortanix-Data-Security-Manager (DSM). SSH is a widely used protocol for securing communication and enabling remote access between systems. It relies on private keys to authenticate users and machines.

With Fortanix DSM, the management and security of these private keys are significantly enhanced. Fortanix DSM stores them in a secure environment and integrates them with PKCS#11 for stronger protection against unauthorized access, streamlining key management, and reducing the risk of credential compromise.

The document demonstrates how to secure the following components in an SSH workflow:

• Client: The client initiates an SSH connection. It contains a private key and uses it to request access to a target host.

• Certificate Authority (CA): The CA uses tools such as ssh-keygen -s to sign client public keys for granting them access to the target host.

• Host: The host runs sshd, which challenges the client's credentials and verifies the client's credentials against its configuration in /etc/ssh/sshd_config.

NOTE

Private keys can also be represented by SSH agents, through ssh-agent and ssh-add.

Additionally, this document explains the setup and configuration of Docker containers to simulate the SSH environment for testing purposes.

2.0 Prerequisites

Ensure the following:

• Linux-based operating system with SSH tools installed. For example, Ubuntu, CentOS.

• Docker installed for setting up the simulated SSH environment.

• PKCS#11 libraries are downloaded and installed on your system. For more information, refer to Clients: PKCS#11 Library.

3.0 Product Tested Version

<>

4.0 Architecture Workflow

<>

5.0 Configure Fortanix DSM

A Fortanix DSM service must be configured, and the URL must be accessible. To create a Fortanix DSM account and group, refer to the following sections:

5.1 Signing Up

To get started with the Fortanix Data Security Manager (DSM) cloud service, you must register an account at <Your_DSM_Service_URL>. For example, https://eu.smartkey.io.

For detailed steps on how to set up the Fortanix DSM, refer to the User's Guide: Sign Up for Fortanix Data Security Manager SaaS documentation.

5.2 Creating an Account

Access the <Your_DSM_Service_URL> on the web browser and enter your credentials to log in to the Fortanix DSM.

5.3 Creating a Group

Perform the following steps to create a group in the Fortanix DSM:

1. Click the Groups menu item in the DSM left navigation panel and click the + button on the Groups page to add a new group.

   

2. On the Adding new group page, enter the following details:

   • Title: Enter a title for your group.

   • Description (optional): Enter a short description for the group.

3. Click the SAVE button to create the new group.

The new group has been added to the Fortanix DSM successfully.

5.4 Creating an Application

Perform the following steps to create an application (app) in the Fortanix DSM:

1. Click the Apps menu item in the DSM left navigation panel and click the + button on the Apps page to add a new app.

   

2. On the Adding new app page, enter the following details:

   • App name: Enter the name of your application.

   • ADD DESCRIPTION (optional): Enter a short description for the application.

   • Authentication method: Select the default API Key as the method of authentication from the drop down menu. For more information on these authentication methods, refer to User's Guide: Authentication documentation.

   • Assigning the new app to groups: Select the group created in Section 5.3: Creating a Group from the list.

3. Click the SAVE button to add the new application.

The new application has been added to the Fortanix DSM successfully.

5.5 Copying an API Key

Perform the following steps to copy the API key from the Fortanix DSM:

1. Click the Apps menu item in the DSM left navigation panel and click the app created in Section 5.4: Creating an Application to go to the detailed view of the app.

2. On the INFO tab, click the VIEW API KEY DETAILS button.

3. From the API Key Details dialog box, copy the API Key of the app to use it later while configuring the Fortanix DSM.

5.6 Creating a Security Object

Perform the following steps to generate a tokenization key in the Fortanix DSM:

1. Click the Security Objects menu item in the DSM left navigation panel and click the + button on the Security Objects page to add a security object.

   

2. On the Add new Security Object page, enter the following details:

   • Security Object name: Enter the name of your security object. For example, my-ssh-ca.

   • Group: Select the group as created in Section 5.3: Creating a Group.

   • Select the GENERATE radio button.

   • Choose a type: Select the RSA key type.

   • Key Size: Indicates the size of the key in bits.

   • Key operations permitted: Select the required operations to define the actions that can be performed with the cryptographic keys.

     NOTE

     Ensure that the keys are configured to permit only Sign and Verify operations, while Encryption and Decryption operations are disabled.

3. Click the GENERATE button to create the new security object.

The new security object is added to the Fortanix DSM successfully.

6.0 Configure Docker Containers for SSH Testing

This section outlines the steps for setting up Docker containers to create a simulated SSH environment for testing. Docker allows you to launch isolated servers, each representing a specific role in the SSH workflow—Client, Certificate Authority (CA), and Server. This approach simplifies testing interactions between these components without the need for physical hardware or complex configurations.

Perform the following steps to configure and simulate the SSH environment using Docker containers:

1. Open a command terminal and run the following command to create a file named docker-compose.yaml:

   <command>

2. Add the following content to the file to define the Docker services:

   version: '3.5'
   services:
     server:
       image: debian:latest
       container_name: server
       ports:
         - "2222:22"
       tty: true
     ca:
       image: debian:latest
       container_name: ca
       tty: true
     client:
       image: debian:latest
       container_name: client
       tty: true

3. Save the file.

4. Run the following command to start the containers:

   docker compose up

5. Run the following command to access the required container:

   docker exec -it <container_name> bash

   Here, replace <container_name> with CA, server, or client based on which container you are configuring.

6. Run the following command to update the package list and install the required services:

   apt update && apt install -y openssh-server rsyslog wget

7. Run the following commands to start the ssh services and check its status:

   service ssh start
   service ssh status

8. Run the following commands to start the rsyslog services and check its status:

   service rsyslog start
   service rsyslog status

9. Run the following command to add a user for SSH access:

   adduser username

   Here, replace the username with any required username that you want to create for SSH access.

10. Run the following command to monitor the authentication log for SSH-related activity:

    tail -f /var/log/auth.log

11. If you are in the client container, run the following command to install the OpenSSH client:

    apt install -y openssh-client

12. Run the following command to test SSH connectivity to the server:

    ssh username@server -p2222 -v

    Here, replace the username with the name of the user you created in Step 9 for SSH access to the server.

If the SSH connection fails, check the server logs in /var/log/auth.log to identify and resolve issues.

7.0 Secure the Target Server with Fortanix DSM

The target server requires a private key to authenticate itself to clients, ensuring that it is the correct machine for the connection. This private key can be securely stored within Fortanix DSM and can only be accessed through an SSH agent running on the server.

If you use this setup, you cannot log in to the server using a password or any other key pairs, as the server is locked to only using one specific private key.

7.1 Configuring the PKCS#11 Interface

Run the following command to download the Fortanix PKCS11 library on the server:

wget https://download.fortanix.com/clients/4.9.2172/fortanix_pkcs11_4.9.2172.so -O /usr/local/lib/sdkms-pkcs11.so

7.2 Fetching the Public Key from Fortanix DSM

Perform the following steps to retrieve the public key from Fortanix DSM:

1. Run the following command to set the API endpoint:

   export FORTANIX_API_ENDPOINT=<YOUR API ENDPOINT>

   Here, replace <YOUR API ENDPOINT> with  the actual URL of your Fortanix DSM API endpoint.

2. Run the following command to generate the public key using the Fortanix PKCS#11 library:

   ssh-keygen -v -D /usr/local/lib/sdkms-pkcs11.so > /etc/ssh/fortanix_host.pub

   When prompted, enter the app API key as copied in Section 5.5: Copying an API key as PIN.

7.3 Adding the Remote Key to an SSH Agent

Perform the following steps to add the remote key to an SSH agent inside the server:

1. Run the following command to start the SSH agent:

   eval ssh-agent

2. Run the following command to add the key to the agent:

   ssh-add -s /usr/local/lib/sdkms-pkcs11.so

   When prompted, enter the app API key  as copied in Section 5.5: Copying an API key as PIN.

   NOTE

   This process may take some time as it checks all PKCS#11 slots.

3. Run the following command to verify the key is added to the agent:

   ssh-add -l

   NOTE

   Note the fingerprint of the displayed key.

7.4 Configuring SSHD to Use the Agent

Run the following command to edit the SSH configuration to use the PKCS#11 key from the agent:

echo "HostKey /etc/ssh/fortanix_host" >> /etc/ssh/sshd_config
echo "HostKeyAgent $SSH_AUTH_SOCK" >> /etc/ssh/sshd_config

NOTE

Ensure to use /etc/ssh/fortanix_host instead of /etc/ssh/fortanix_host.pub.

7.5 Reloading SSHD and Checking SSH Access

Perform the following steps to reload sshd and check the ssh access from the client:

1. Run the following commands to restart the SSH service on the server:

   service ssh stop && service ssh start

2. Run the following command to check the status of SSH service on the server:

   service ssh status

3. Run the following command to test SSH access from the client:

   ssh username@server

   NOTE

   The client should display a warning if the key has changed. Compare the fingerprints displayed on the client output with the agent key fingerprint.

4. Run the following command on the server to view the agent key:

   ssh-add -l

7.6 Disabling the RSA Key in Fortanix DSM

Perform the following steps to disable the RSA key:

1. Navigate to Security Objects menu item from the DSM left navigation panel and click the RSA key created in Section 5.6: Creating a Security Object.

2. In the detailed view of the security object, scroll to the end of the page and click DEACTIVATE NOW button to disable the RSA key.

   

7.7 Checking the SSH Access

<>

8.0 Secure the CA Server Keys with Fortanix DSM (SSH Certificate Authentication)

SSH certificate-based authentication simplifies the process of managing authorized clients across multiple hosts.

Instead of maintaining a list of authorized SSH clients, for example: ~/.ssh/authorized_keys, SSH certificates can be signed by a trusted CA and used for authentication.

The host checks the following criteria during SSH authentication to determine if the access should be granted:

• Trusted CA: The host checks if the certificate was signed by a CA it trusts. The trusted CA specified in the sshd_config file using TrustedUserCAKeys /etc/ssh/ca.pub.

• Certificate Validity and Revocation: The host verifies whether the certificate is valid and has not been revoked. This is done by checking a Certificate Revocation List (CRL), which can be maintained by the CA.

• Signature Verification: The host checks the certificate’s signature to ensure it is authentic.

• Principal List: The certificate contains a list of principals (users), and the host checks if at least one principal matches an authorized user. This is configured in AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u.

• Private Key Ownership: The client must have the private key corresponding to the public key in the certificate they are attempting to authenticate with. This way only the rightful owner of the private key can use the certificate.

To implement SSH certificate authentication and protect the CA server's keys, Fortanix DSM can be used to store the private keys of both the CA and client securely. By doing so, the security of the CA keys is enhanced, ensuring that private keys are not exposed during the signing of certificates.

8.1 Creating a Client Key Pair

Run the following command to generate an SSH key pair using ssh-keygen with the ed25519 algorithm. The private key will be saved as ./cli and the public key as ./cli.pub:

./ssh-keygen -t ed25519 -C "<email_id>"

Enter a passphrase when prompted (strongly recommended) for added security. The key pair will be saved in the current directory as cli (private key) and cli.pub (public key).

8.2 Preparing the Fortanix Shared Object and Creating the CA Key

Perform the following steps to prepare the Fortanix shared object and create the CA key:

1. Download the PKCS#11 shared object from Fortanix's platform and save it locally as fortanix_pkcs11_<version>.so. For more information, refer to Clients: PKCS#11 Library.

2. Create the pkcs11.conf file and add the following content:

   api_key = "<YOUR API KEY>"
   api_endpoint = "<YOUR API ENDPOINT>"
   [log]
   file = "<A LOG FILE OF YOUR CHOICE>"

3. Run the following command to verify that ssh-keygen detects the shared object and retrieves the public key corresponding to my-ssh-ca:

   ssh-keygen -D fortanix_pkcs11.so

   When prompted for the PIN, provide the path to the configuration file as file://pkcs11.conf. This should return the public key for my-ssh-ca.

4. Run the following command to save the output public key. For example, ca.pub.

   ./ssh-keygen -D fortanix_pkcs11.so > ca.pub

8.3 Preparing the Server for SSH Certificate Authentication

Perform the following steps to prepare the server for SSH certificate authentication:

1. Run the following command to create a user on the target host for SSH authentication:

   adduser username

2. Run the following command to copy the CA public key (ca.pub) to the server's /etc/ssh/ca.pub file. You can have multiple CAs here, one per line.

   sudo cp ca.pub /etc/ssh/ca.pub

3. Run the following commands to create the principals file and populate it with the intended authorized principals:

   mkdir -p /etc/ssh/auth_principals/
   echo "username" > /etc/ssh/auth_principals/username

4. Run the following command on the host to modify the sshd_config file to specify the required settings for SSH certificate authentication:

   # So that the SSH server asks for the public keys of the client.
   PubKeyAuthentication yes
   
   # The certificate MUST carry valid principals for correct authentication
   AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
   
   # Disallow password auth.
   PasswordAuthentication no
   
   # Specifies which CAs to trust.
   TrustedUserCAs /etc/ssh/ca.pub
   
   # Revocations
   RevokedKeys /etc/ssh/revoked

5. Run the following command to restart the SSH service:

   service ssh restart

8.4 Signing the Client Public Key

Perform the following steps to sign the client public key with the CA:

1. Run the following command to sign the client public key (cli.pub) using the ssh-keygen tool, specifying the CA's public key (ca.pub), validity period, and principal:

   ./ssh-keygen -D fortanix_pkcs11.so -s ca.pub -V "-1m:+4m" -I "<email_id>" -n "username" cli.pub

   This command generates a signed certificate cli-cert.pub.

2. Run the following command to inspect the signed certificate:

   ssh-keygen -L -f cli-cert.pub

   This outputs the certificate details such as the key ID, serial number, valid time, and authorized principals.

   li-cert.pub:
           Type: [email protected] user certificate
           Public key: ED25519-CERT SHA256:1G7qby3LkUdTsuGLF/vhljgfo5hDaFUCTKhpwm79unY
           Signing CA: RSA SHA256:Uiz1MhCWgrhpCNy6KB5DzRseH8bmUMZA0ZiRiNnBgrg (using rsa-sha2-512)
           Key ID: "<email_id>"
           Serial: 0
           Valid: from 2021-09-13T12:25:33 to 2021-09-13T12:30:33
           Principals: 
                   username
           Critical Options: (none)
           Extensions: 
                   permit-X11-forwarding
                   permit-agent-forwarding
                   permit-port-forwarding
                   permit-pty
                   permit-user-rc

8.5 Logging in to the Server Using SSH

Perform the following steps to log in to the server using SSH:

Run the following command to SSH to the server using the signed client certificate:

ssh -i ./cli username@host -p 2222 -v

The client will use the cli private key and cli-cert.pub certificate for authentication.

OpenSSH_8.2p1 Ubuntu-4ubuntu0.3, OpenSSL 1.1.1i  8 Dec 2020
debug1: Reading configuration data /home/<user>/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to localhost [127.0.0.1] port 2222.
debug1: Connection established.
debug1: identity file ./cli type 3
debug1: identity file ./cli-cert type 7
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2
debug1: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to localhost:2222 as 'username'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:gSQm0niyUYqW3C1Mlpd3EeoALdZ/tZ25xh+6SD/hux0
debug1: Host '[localhost]:2222' is known and matches the ECDSA host key.
debug1: Found key in /home/<user>/.ssh/known_hosts:28
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug1: Will attempt key: ./cli ED25519 SHA256:iRi5zdsSNkCTRPqUyS4HZzK6QTc4ZTx9GLMurFywrc4 explicit
debug1: Will attempt key: ./cli ED25519-CERT SHA256:iRi5zdsSNkCTRPqUyS4HZzK6QTc4ZTx9GLMurFywrc4 explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: ./cli ED25519 SHA256:iRi5zdsSNkCTRPqUyS4HZzK6QTc4ZTx9GLMurFywrc4 explicit
debug1: Authentications that can continue: publickey
debug1: Offering public key: ./cli ED25519-CERT SHA256:iRi5zdsSNkCTRPqUyS4HZzK6QTc4ZTx9GLMurFywrc4 explicit
debug1: Server accepts key: ./cli ED25519-CERT SHA256:iRi5zdsSNkCTRPqUyS4HZzK6QTc4ZTx9GLMurFywrc4 explicit
Enter passphrase for key './cli': 
debug1: Authentication succeeded (publickey).
Authenticated to localhost ([127.0.0.1]:2222).
debug1: channel 0: new [client-session]
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug1: Remote: cert: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: principals: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: cert: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: principals: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Linux f268b80e896c 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC 2021 x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Aug 26 08:40:31 2021 from 172.28.0.1

After a successful authentication, you will be logged into the server.

9.0 Revocation of a Certificate

This section explains how to handle certificate revocation and maintain a Certificate Revocation List (CRL) or Key Revocation List (KRL).

Use the ssh-keygen tool to add the revoked certificate to a revocation list (KRL). Run the following command to revoke a certificate:

ssh-keygen -k -f /etc/ssh/revoked cli-cert.pub

Where,

• -k creates a revocation list (KRL).

• -f /etc/ssh/revoked specifies the location where the revocation list file is stored.

• cli-cert.pub is the certificate to be revoked.

If you need to add more certificates to the same list, use the -u flag after -k. This will update the KRL without replacing the existing entries.

ssh-keygen -k -f /etc/ssh/revoked -u cli-cert2.pub

Where, -u adds the specified certificate to the existing KRL.

10.0 Secure the Client Key with Fortanix DSM

To secure the client key, you must create a new group, application, and an RSA security object in Fortanix DSM with SIGN and VERIFY permissions. For detailed steps, refer to Section 5.0: Configure Fortanix DSM.

10.1 Creating and Editing cli.conf file

Perform the following steps to create and edit the cli.conf file:

Create a cli.conf configuration file to store your app API credentials and logging details. Add the following content in the cli.conf file:

api_key = "<YOUR API KEY>"
api_endpoint = "<YOUR API ENDPOINT>"
[log]
file = "<A LOG FILE OF YOUR CHOICE>"

Where,

• <YOUR API KEY> refers to the API key copied from the Fortanix DSM app.

• <YOUR API ENDPOINT> refers to the endpoint URL for the Fortanix DSM API.

• <A LOG FILE OF YOUR CHOICE> refers to the path to the log file where you want to store logs for operations.

This file should be placed securely and referenced when interacting with the Fortanix PKCS#11 library.

10.2 Extracting the Public Key of the Client

Perform the following steps to extract the public key of the client:

You can extract the client's public key using the ssh-keygen command to interact with the Fortanix PKCS#11 library.

Run the following command to extract the public key:

ssh-keygen -D fortanix_pkcs11.so > cli.pub

When prompted, enter the PIN for the Fortanix Token, which is set in the cli.conf file.

Run the following command to check the contents of the public key:

cat cli.pub

The output may appear as follows:

ssh-rsa AAAAB3NzaC1yc2...TRUNCATED...G0WQCN8Lq9 ssh-cli

10.3 Logging in to a Target Server Using the Client Key

To use the secured client key and log in to a remote server, you need to specify the Fortanix PKCS#11 shared object and, provide the signed certificate, if needed.

Run the following SSH command to authenticate using the client key:

ssh -I fortanix_pkcs11.so -i cli.pub username@server -p <PORT> -v

Where,

• -I fortanix_pkcs11.so: Specifies the Fortanix PKCS#11 shared object, which contains the private key used for signing.

• -i cli.pub: Specifies the public key, which is signed by the private key in the shared object.

• username@server: Replace with the target server's username and IP address/hostname.

• -p <PORT>: Replace this with the correct SSH port.

• -v: This flag shows detailed information to help debug any connection issues.