---
title: "Fortanix DSM with Databricks"
slug: "fortanix-dsm-with-databricks"
updated: 2026-07-27T17:08:19Z
published: 2026-07-27T17:08:19Z
canonical: "support.fortanix.com/fortanix-dsm-with-databricks"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://support.fortanix.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Fortanix DSM with Databricks

## 1.0 Introduction

This article describes how to integrate **Fortanix-Data-Security-Manager (DSM)** with **Databricks** to enhance data security measures within the Databricks SQL warehouse. The primary goal of this integration is to enforce robust data security protocols, ensuring the confidentiality of sensitive data stored in the warehouse.

Tokenization and detokenization are fundamental techniques in data security, enabling the protection of sensitive data while preserving its usability.

## 2.0 Overview

Within this integration framework, the Fortanix DSM's vaultless tokenization is used with Databricks Notebooks and Python user-defined functions (UDFs) to facilitate tokenization and detokenization processes within the Databricks SQL warehouse.

- **Method 1 - Databricks Notebooks**: These are essential tools for data science and machine learning workflows. They offer real-time co-authoring, automatic versioning, and built-in data visualizations. In this approach, Fortanix DSM Python Software Development Kits (SDKs) are used to connect to Fortanix DSM API endpoints within Databricks Notebooks to tokenize and detokenize sensitive data stored in the SQL warehouse. This ensures the secure transformation of sensitive data columns in the table.

> [!NOTE]
> NOTE
> 
> Notebooks support both tokenization and detokenization operations.

*For more information, refer to* [*Introduction to Databricks Notebooks*](https://docs.databricks.com/en/notebooks/index.html)*.*
- **Method 2 - Python UDFs**: These allow for secure and governed execution of Python code through SQL functions. By integrating UDFs with the Fortanix DSM API, you can perform tokenization operations to redact email and phone information from JSON strings, returning the redacted string and preventing unauthorized access.

> [!NOTE]
> NOTE
> 
> Python UDFs are currently in public preview mode and support both tokenization and detokenization operations.

*For more information, refer to* [*User-defined functions (UDFs) in Unity Catalog*](https://docs.databricks.com/en/udf/unity-catalog.html)*.*

## 3.0 Prerequisites

Ensure the following:

- You must have a premium subscription account in the Databricks application.
- To use UDFs, Python UDFs must be enabled. These are currently in preview mode.
  - [*Databricks Documentation on UDF*](https://docs.databricks.com/en/udf/unity-catalog.html)
  - [*Blog Post: Introducing Python UDFs in Databricks SQL*](https://www.databricks.com/blog/2022/07/22/power-to-the-sql-people-introducing-python-udfs-in-databricks-sql.html)
- Fortanix DSM must be accessible. *For more information, refer to* [*Section 6.1: Signing Up*](/v1/docs/using-data-security-manager-with-databricks#61-signing-up) and [*Section 6.2: Creating an Account*](/v1/docs/using-data-security-manager-with-databricks#62-creating-an-account).

## 4.0 Product Tested Version

This integration has been tested on the following versions:

- Fortanix DSM version 4.30 and above.

## 5.0 Architecture Diagram

![](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_Architecture_Diagram.png)

**Figure 1: Architecture diagram**

The integration architecture is divided into two main planes: the **Databricks Control Plane** and the **Databricks Data Plane**, both connected to Fortanix DSM for secure data operations.

The **Databricks Control Plane** manages credentials for connecting to Fortanix DSM APIs and is where SQL commands for operations on sensitive data are written. It handles configuration, authentication, and orchestration.

The **Databricks Data Plane** is where the data resides and undergoes processing. It includes resources like SQL warehouses and data lakes. The **Databricks Data Plane** executes the SQL commands for tokenizing and detokenizing sensitive data.

The integration flow begins in the **Databricks Control Plane**, where Notebooks or Python UDFs use the Fortanix DSM Python SDK to connect to Fortanix DSM. Then, the SQL commands to tokenize and detokenize sensitive data in the Databricks SQL warehouse are written in the Control Plane and executed in the Data Plane.

## 6.0 Configure Fortanix DSM

A Fortanix DSM service must be configured, and the URL must be accessible. To create a Fortanix DSM account and group, refer to the following sections:

### 6.1 Signing Up

To get started with the Fortanix DSM cloud service, you must register an account at <Your_DSM_Service_URL>. For example, [https://amer.smartkey.io.](https://amer.smartkey.io.) On-premises customers use the KMS URL, and the SaaS customers can use the URLs as listed [*here*](https://support.fortanix.com/hc/en-us/articles/4406135346068-Fortanix-DSM-SaaS-Global-Availability-Map) based on the application region.

*For more information on how to set up the Fortanix DSM, refer to the* [*Sign Up for Fortanix Data Security Manager SaaS*](https://support.fortanix.com/docs/users-guide-sign-up-for-fortanix-data-security-manager-saas)*.*

### 6.2 Creating an Account

Access <Your_DSM_Service_URL> in a web browser and enter your credentials to log in to Fortanix DSM.

![](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/DSM_SaaS_Login_page(15).png)

**Figure 2: Logging in**

*For more information on how to set up an account in Fortanix DSM, refer to the* [*Getting Started with Fortanix Data Security Manager - UI*](https://support.fortanix.com/docs/users-guide-getting-started-with-fortanix-data-security-manager-ui)*.*

### 6.3 Creating a Group

Perform the following steps to create a group in the Fortanix DSM:

1. In the DSM left navigation panel, click the **Groups** menu item, and then click **ADD GROUP** to create a new group.

![](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Add-Group(7).png)

**Figure 3: Add groups**
2. On the **Adding new group** page:

**Title**: Enter a name for your group.
  1. **Description** (optional): Enter a short description of the group.
3. Click **SAVE** to create the new group.

*For more information about groups, refer to* [*Definitions*](/v1/docs/dsm-definitions#40-groups)*.*

> [!NOTE]
> TIP
> 
> It is recommended to configure a **Key undo policy** for groups for the cryptographic keys used by the application before creating or using the keys. Without a configured **Key undo policy**, key deletion is permanent and the deleted key cannot be recovered. Deleting an encryption key may result in permanent loss of access to the protected data and service disruption for applications that depend on the deleted key. *For detailed steps, refer to* [*Key Undo Policy*](/v1/docs/fortanix-dsm-key-undo-policy)*.*

### 6.4 Creating an Application

Perform the following steps to create an application (app) in the Fortanix DSM:

1. In the DSM left navigation panel, click the **Apps** menu item, and then click **ADD APP** to create a new app.

![](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Add-App(8).png)

**Figure 4: Add application**
2. On the **Adding new app** page:
  1. **App name**: Enter the name for your application.
  2. **ADD DESCRIPTION** (optional): Enter a short description of the application.
  3. **Authentication method**: Select the default **API Key** as the authentication method from the drop down menu. *For more information on these authentication methods, refer to the* [*User's Guide: Authentication*](https://support.fortanix.com/docs/users-guide-authentication)*.*
  4. **Assigning the new app to groups**: Select the group created in [*Section 6.3: Creating a Group*](/v1/docs/en/using-data-security-manager-with-databricks#63-creating-a-group)[](/v1/docs/using-data-security-manager-with-idcentral-key-management#43-creating-a-group)from the list.
3. Click **SAVE** to add the new application.

*For more information about applications, refer to* [*Definitions*](https://support.fortanix.com/docs/dsm-definitions#50-applications)*.*

### 6.5 Copying the API Key

Perform the following steps to copy the API key from the Fortanix DSM:

1. In the DSM left navigation panel, click the **Apps** menu item, and then click the app created in [*Section 6.4: Creating an Application*](/v1/docs/en/using-data-security-manager-with-databricks#64-creating-an-application) [](/v1/docs/using-data-security-manager-with-idcentral-key-management#44-creating-an-application)to go to the detailed view of the app.
2. On the **INFO** tab, click **VIEW API KEY DETAILS**.
3. From the **API Key Details** dialog box, copy the **API Key** of the app to be used later.

### 6.6 Creating a Security Object

> [!NOTE]
> NOTE
> 
> For this guide, two tokenization security objects are created to tokenize and detokenize the Last Name and Email Address.

Perform the following steps to generate a tokenization key in the Fortanix DSM:

1. In the DSM left navigation panel, click the **Security Objects** menu item, and then click **ADD SECURITY OBJECT** to create a new security object.

![](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Add-SO(4).png)

**Figure 5: Adding security object**
2. On the **Add new Security Object** page:
  1. **Security Object Name**: Enter the name of your security object. For example, **db_name_token**.
  2. **Group**: Select the group as created in [*Section 6.3: Creating a Group*](/v1/docs/en/using-data-security-manager-with-databricks#63-creating-a-group).
  3. Select **GENERATE**.
  4. In the **Choose a type** section, select the **Tokenization** key type to generate.
  5. In the **Key Size** section, select the size of the key in bits
  6. In the **Data type** section, select the type of the security object token. *For more information, refer to the* [*User's Guide: Tokenization*](/v1/docs/users-guide-tokenization)*.*
  7. In the **Key operations permitted**, section select the required operations to define the actions that can be performed with the cryptographic keys, such as encryption, decryption, signing, and verifying.
3. Click **GENERATE** to create the new security object.
4. Similarly, repeat the *Steps 1 to 3* to create a security object for **Email Address** as well. For example, **db_email_token**.

![](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Tokenization-SO-Sceen.png)

**Figure 6: Security objects added**

*For more information about security objects, refer to* [*Definitions*](https://support.fortanix.com/docs/dsm-definitions#60-security-objects)*.*

## 7.0 Create a Databricks Secret

You can use the Databricks [Secret Management](https://docs.databricks.com/en/security/secrets/index.html) to store the Fortanix DSM API key, rather than hardcoding it in Notebooks or Python UDFs to ensure secure and authorized access to the API keys for Notebooks and UDFs at runtime.

Perform the following steps using Databricks CLI:

1. Run the following commands to create a Databricks secrets scope:

```bash
SECRETS_SCOPE_NAME="<your secret scope name>"
                    databricks secrets create-scope $SECRETS_SCOPE_NAME
```
2. Run the following commands to list the scopes:

```bash
databricks secrets list-scopes
```

The output of the command will be:

```bash
Scope Backend Type
                        hr_scope DATABRICKS
```
3. Run the following commands to add users to the scope to access the secret:

```bash
databricks secrets put-acl $SECRETS_SCOPE_NAME adam.hr@example.com MANAGE
```
4. Run the following commands to view the list of users who have the access to the secret:

```bash
databricks secrets list-acls $SECRETS_SCOPE_NAME
```

The output of the command will be:

```bash
[
                        {
                        "permission":"MANAGE",
                        "principal":"adam.hr@example.com"
                        }
                        ]
```
5. Run the following commands to add a secret:

```bash
databricks secrets put-secret $SECRETS_SCOPE_NAME FORTANIX_API_KEY --string-value "<API_KEY_VALUE>"
```
6. Run the following commands to view the list of the secrets:

```bash
databricks secrets list-secrets $SECRETS_SCOPE_NAME
```

The output of the command will be:

```bash
Key Last Updated Timestamp
                        FORTANIX_API_KEY 1714076216500
```
7. Run the following commands to get the value of the secret:

```bash
databricks secrets get-secret $SECRETS_SCOPE_NAME FORTANIX_API_KEY
```

The output of the command will be:

```bash
{
                        "key":"FORTANIX_API_KEY",
                        "value":"<base64_encoded_value>"
                        }
```
8. Run the following commands to get `base64` decoded value of the secret:

```bash
databricks secrets get-secret $SECRETS_SCOPE_NAME FORTANIX_API_KEY | jq -r '.value' | base64 -d
```

*For more information, refer to* [*Access Control List*](https://docs.databricks.com/en/security/auth-authz/access-control/index.html#secrets)*.*

## 8.0 Methods

This section elaborates the steps for integrating either of the two Databricks methods as defined in [*Section 2.0: Overview*](/v1/docs/using-data-security-manager-with-databricks#20-overview).

### 8.1 Using Databricks Notebooks

Perform the following steps to create a new Databricks Notebook to import Fortanix DSM Python SDK, and define the tokenization and detokenization functions:

1. Log in to your Databricks Secret Management account.
2. From the left navigation panel, click **+ NEW** → **Notebook** to create a new Notebook.

![databricks_add_new_notebook](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_New_Notebook.png)

**Figure 7: Add notebook**
3. On the next screen, click **File** tab → **Import notebook…** from the drop down menu to import a sample DSM Notebook.

![databricks_import_notebook](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_Import_Notebook.png)

**Figure 8: Import notebook**
4. On the **Import** dialog box,
  1. **Import from**: Select **URL**.
  2. **Target Folder**: The path to the target folder is selected by default.
  3. **URL**: Enter the following URL in the provided space: [*https://github.com/fortanix/databricks/blob/main/notebook/DSM_notebook.p*y](https://github.com/fortanix/databricks/blob/main/notebook/DSM_notebook.py).

![dtatabricks_import_dialog_box](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_Import_URL.png)

**Figure 9: Import dialog box**
5. Click **Import** to import the sample DSM Notebook file. This Notebook fetches the API key from the Databricks secrets as configured in [*Section 7.0: Creating a Databricks Secret*](/v1/docs/using-data-security-manager-with-databricks#70-creating-a-databricks-secret).

```bash
api_key = dbutils.secrets.get(scope="hr_scope", key="FORTANIX_API_KEY")
```

Where,
  - `scope`: Refers to the secret scope name defined in the Databricks Secret Management.
  - `FORTANIX_API_KEY`: Refers to the Fortanix DSM API key of the app copied in [*Section 6.5: Copying an API Key*](/v1/docs/using-data-security-manager-with-databricks#65-copying-an-api-key).
6. Click **Run cell** to run the Python script in the Notebook to validate if everything works.

![databricks_run_button](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_Notebook1_Run.png)

**Figure 10: Run**
7. In the **Attach to an existing compute resource** form, select **General compute** and click **Start, attach and run** to execute the Notebook. Wait for a few minutes to validate the status of the Notebook. The Notebook will be created with the name **DSM_notebook**.

After the status of the Notebook is validated, you need to create a second Notebook in the account to define the cryptographic keys UUIDs and call `tokenize_col` and `detokenize_col` as defined in the previous steps.

Perform the following steps:

1. From the left navigation panel, click **+ NEW** → **Notebook**.

![databricks_add_notebook1](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_New_Notebook2.png)

**Figure 11: Add notebook**
2. On the next screen, click **File** → **Import notebook…** from the drop down menu.

![databricks_import_notebook1](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_Import_Notebook2.png)

**Figure 12: Import notebook**
3. In the **Import** dialog box:
  1. **Import from**: Select **URL**.
  2. **Target Folder**: The path to the target folder is selected by default.
  3. **URL**: Enter the following URL in the provided space: [*https://github.com/fortanix/databricks/blob/main/notebook/tokenize_and_detokenize_sample_notebook.py*](https://github.com/fortanix/databricks/blob/main/notebook/tokenize_and_detokenize_sample_notebook.py).
4. Click **Import** to import the new sample DSM Notebook file. This Notebook references the **DSM_notebook** notebook as created in the previous section. It defines the key UUID values, and invokes the tokenization (`tokenize_col`) and detokenization (`detokenize_col`) functions described as follows: These functions expect the table name, column name, and Key UUIDs as inputs.
  1. Run the following command to source the Fortanix DSM Notebook:

```bash
%run "./DSM_notebook"
```
  2. Run the following command to define the Fortanix Key UUIDs:

```bash
lname_kid = "<LAST_NAME_KEY_UUID>"
                            email_kid = "<EMAIL_ADDRESS_KEY_UUID>"
```

Where,
    - `lname_kid`: Refers to the Fortanix DMS key UUID for the last name key as created in Section 6.6: Creating a Security Object.
    - `email_kid`: Refers to the Fortanix DSM key UUID for the email key as created in [*Section 6.6: Creating a Security Object*](/v1/docs/using-data-security-manager-with-databricks#66-creating-a-security-object).
  3. In this sample notebook, the pre-created table has the following structure:

```bash
# "employees.data"
                            # employee_id bigint
                            # fname string
                            # lname string
                            # email string
```

![databricks_employees_table](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_EnployeesData_Table.png)

**Figure 13: Employees data table**
  4. Run the following command to tokenize the columns:

```bash
tokenize_col("employees","data",["lname","email"],[lname_kid,email_kid])
                            # Comment out calling insert_tokenizedData() from tokenize_col() in DSM_notebook if you not need another table to be created.
```

This sample function tokenizes the `lname` and `email` columns and creates a new table called `tokenized_&lt;table_name&gt;`.
  5. Run the following command to detokenize the columns by specifying the table and column names:

```bash
detokenize_col("employees","tokenized_data",["lname","email"],[lname_kid,email_kid])
```
5. Click **Run cell** to run the Python script for the second Notebook.

![databricks_sample_notebook](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_Tokenize_Detokenize.png)

**Figure 14: Sample notebook**
6. In the **Attach to an existing compute resource** form, select **General compute** and click **Start, attach and run** to execute the following functions in the Notebook:
  - Tokenize the selected columns, such as, `lname`, `email`.
  - Create a new table named `employeed.tokenized_data` in the Catalog.
  - Detokenize the same columns from table `employeed.tokenized_data`.

![databricks_tokenized_table](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_EnployeesData_Tokenized_Table.png)

**Figure 15: Employees tokenized data table**

### 8.2 Using Python User-defined functions

Perform the following steps to implement tokenization and detokenization operations in Databricks SQL warehouse using Python UDFs:

> [!NOTE]
> NOTE
> 
> Ensure that the Python UDF is enabled, which is currently in public preview.

1. From the left navigation panel, click **SQL Editor**.
2. In the **New query** working space, paste the content from the [Tokenization UDF Function](https://github.com/fortanix/databricks/blob/main/python_udf/tokenizie_udf_unity.py) file.
3. Click the run icon to execute the function file.

![databricks_run-button1](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_NewQuery_Run.png)

**Figure 16: Run**

The results are displayed in the **Raw results** section.
4. After the executing of the script is completed, click **+** to add a new query.

![databricks_create_query](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_Create_NewQuery.png)

**Figure 17: Create new query**
5. Similarly, in the **New query** working space, paste the content from the [Detokenization UDF Function](https://github.com/fortanix/databricks/blob/main/python_udf/detokenizie_udf_unity.py) file.
6. Click the run icon to execute the function file.

![databricks_new_query](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_NewQuery2_Run.png)

**Figure 18: Run**
7. Create a new query to tokenize the email column of the `employees.data` table available in the Catalog. Copy and paste the following commands to tokenize the email column:

```bash
SELECT fortanix_tokenize(
                    email,
                    map(
                    'fortanix_api_endpoint', 'https://apac.smartkey.io',
                    'fortanix_api_key', secret('hr_scope', 'FORTANIX_API_KEY'),
                    'key_id', 'fdc25a25-f75b-4a5a-83d6-002c8ed71fba'
                    )
                    ) from data;
```

Where,
  - `fortanix_api_endpoint`: Refers to the URL endpoint for the Fortanix DSM API.
  - `fortanix_api_key`: Refers to the Fortanix DSM API key of the app copied in [*Section 6.5: Copying an API Key*](/v1/docs/using-data-security-manager-with-databricks#65-copying-an-api-key).
  - `secret`: Refers to the secret scope name defined in the Databricks Secret Management.
  - `key_id`: Refers to the unique identifier of the encryption key used for tokenization.
8. Click **Run** to execute the Python query to view the results of tokenized email column in the **Raw results** section:

![databricks_tokenize_query](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_TokenizeQuery.png)

**Figure 19: Tokenized query**
9. Create a new query to detokenize the email column of the `employees.tokenized_data` table available in the Catalog. Copy and paste the following commands to detokenize the email column:

```bash
SELECT fortanix_detokenize(
                    email,
                    map(
                    'fortanix_api_endpoint', 'https://apac.smartkey.io',
                    'fortanix_api_key', secret('hr_scope', 'FORTANIX_API_KEY'),
                    'key_id', 'fdc25a25-f75b-4a5a-83d6-002c8ed71fba'
                    )
                    ) from tokenized_data;
```

Where,
  - `fortanix_api_endpoint`: Refers to the URL endpoint for the Fortanix DSM API.
  - `fortanix_api_key`: Refers to the Fortanix DSM API key of the app copied in [*Section 6.5: Copying an API Key*](/v1/docs/using-data-security-manager-with-databricks#65-copying-an-api-key).
  - `key_id`: Refers to the unique identifier of the encryption key used for tokenization.
10. Click **Run** to execute the Python query to view the results of detokenized email column in the **Raw results** section:

![](https://cdn.us.document360.io/c3bd85d2-4ad8-4d85-9f60-f1c168a3aad9/Images/Documentation/Databricks_DetokenizeQuery(1).png)

**Figure 20: Detokenized query**

Fortanix Data Security Manager (DSM) is the world’s first cloud service secured with Intel® SGX. With Fortanix DSM, you can securely generate, store, and use cryptographic keys and certificates, as well as other secrets such as passwords, API keys, tokens, or any blob of data. Your business-critical applications and containers can integrate with Fortanix DSM using legacy cryptographic interfaces (PKCS#11, CNG, and JCE) or using the native Fortanix DSM RESTful interface.