Workflow Applications Using Fortanix EDP

1.0 Introduction

Workflow graphs are maps that show how generic applications are connected to datasets and other generic applications. These are collaborative objects where multiple users can provide their own objects and approvals. For more information, refer to Create, Update, Clone, and Delete Workflows.

This article describes how to create and deploy Fortanix CCM Workflows using a simple data processing example.

1.1 A Research Portal for Patient Data

This example demonstrates a Fortanix CCM workflow with Rust applications deployed securely using Fortanix EDP.

The Workflow provides stratification of patient data by demographic attributes (for example: age, gender, ethnicity).

For input data, a synthetic file containing the medical history of fictitious patients is used.

The data used for this example is available from https://synthea.mitre.org/downloads.

File: 1K Sample Synthetic Patient Records, CSV[mirror]: 9 MB

This is made available via:
Jason Walonoski, Mark Kramer, Joseph Nichols, Andre Quina, Chris Moesel, Dylan Hall, 
Carlton Duffett, Kudakwashe Dube, Thomas Gallagher, Scott McLachlan, 
Synthea: An approach, method, and software mechanism for generating synthetic patients 
and the synthetic electronic health care record, Journal of the American Medical 
Informatics Association, Volume 25, Issue 3, March 2018, 
Pages 230–238, https://doi.org/10.1093/jamia/ocx079

The input and output datasets represent patient information collected by a different healthcare provider and contain the following elements

Patient demographic information
Patient conditions
Patient encounters
Patient medications

2.0 Create CCM Workflow

2.3 Select the Account

Create or select a Fortanix Armor account. For detailed instructions, refer to Getting Started with Fortanix Armor.

2.4 Create a Group

Create a group in the Fortanix Armor Identity and Access Management solution. For detailed instructions, refer to Fortanix Armor Identity and Access Management.

2.5 Invite Users to Join Fortanix Armor

An Administrator invites users who are Data Owners, App Owners, and Output Owners to join a Fortanix Armor account and create a workflow graph in Fortanix CCM.

For detailed instructions on inviting users, refer to Fortanix Armor Identity and Access Management.

The new users (Data Owners and App Owners) must join the Fortanix Armor account and create Datasets and App Configs as described in the following sections.

2.6 Writing the Application

Fortanix EDP applications are Rust applications using the Rust-SGX SDK.

For the EDP application, the user needs to write the application to perform operations such as downloading and decrypting the patient’s input conditions data, processing it, and encrypting and uploading the output conditions data to a secure AWS S3 location.

This functionality is available in the Fortanix repository mentioned in Bringing EDP Rust Apps to Confidential Computing Manager.

The following example is used to process the CSV datasets.

fn process(decrypted: Vec<u8>) -> Result<String, String> {
    [allow(non_snake_case)]
    [derive(Debug, Deserialize)]
    struct Record {
        START: String,
        STOP: String,
        PATIENT: String,
        ENCOUNTER: String,
        CODE: String,
        DESCRIPTION: String,
    };
    let mut statistics = HashMap::<String, u32>::new();
    let mut count : u32 = 0;
    let mut rdr = csv::Reader::from_reader(&*decrypted);
    for i in rdr.deserialize() {
        let record: Record = i.map_err(|_| "Invalid CSV data".to_string())?;
        statistics.entry(record.DESCRIPTION).and_modify(|e| *e += 1).or_insert(1);
        count += 1;
    }
   
    let last_entry = statistics.iter().max_by(|a, b| a.1.cmp(&b.1)).ok_or("No entries in CSV")?;
    let top = last_entry.0;
    let freq = last_entry.1;
    let unique = statistics.len();
   
    let result = format!("count {:<50}\nunique {:<50}\ntop {:<50}\nfreq {:<50}\nName: DESCRIPTION, dtype: object\n",
    count, unique, top, freq);
   
    Ok(result)
}

3.0 Sign the Application and Create Docker Image

Next, the user must sign the application using a development certificate:

cargo build --target=x86_64-fortanix-unknown-sgx
app_bin=./target/x86_64-fortanix-unknown-sgx/debug/harmonize

signing_cert=$(dirname ${app_bin})/private.pem
openssl genrsa -3 3072 > ${signing_cert}

ftxsgx-elf2sgxs ${app_bin} --heap-size 10240000 --stack-size 10240000 --threads 5 --debug
sgxs-sign ${app_bin}.sgxs "${app_bin}.sigstruct" --key ${signing_cert} -p 1 -v 1

You can also build a Docker image using the following Docker file. It copies harmonize.sgxs/ftxsgx-runner to the current folder:

FROM ubuntu:20.04
RUN apt-get update && apt-get install -y --no-install-recommends libssl-dev
WORKDIR /root/
COPY ./harmonize.sgxs .
COPY ./ftxsgx-runner .
ENTRYPOINT ["/root/ftxsgx-runner", "/root/harmonize.sgxs"]

The parameters for registering the application are in the file ./target/x86_64-fortanix-unknown-sgx/debug/harmonize.sigstruct created by the sign operation.

4.0 Create an EDP Application Using UI

Perform the following steps:

To create an EDP application using Fortanix CCM UI:
1. Navigate to the Applications menu item in the CCM UI left navigation panel and click ADD APPLICATION to create a new application.
2. Now, in the Add application dialog box, select EDP application, and click NEXT.
3. In the Add application form, add the details of the application and click ADD APPLICATION to complete creating the application.
For more information, refer to Add EDP Application.
Create a build of the application by uploading the SIGSTRUCT file (or pasting the base64 version of it). The SIGSTRUCT used in this example is here.
For more information, refer to Create Application Build for EDP Applications.
Fetch the build whitelisting tasks using the Fortanix CCM Tasks tab.
Approve the task.

5.0 Create Input and Output Datasets

As input and output datasets to this workflow, two synthetic datasets are used that contain the medical history of synthetic patients.

In this example, create the following data points and credentials for download/upload:

AWS S3 URL for Input dataset:
https://fortanix-pocs-data.s3.amazonaws.com/conditions.csv.enc
AWS S3 URL for Output dataset:
https://fortanix-pocs-data.s3.amazonaws.com/conditions_output.csv.enc
Associated encryption keys and access credentials: ./key.hex

5.1 Input User

Consider that the Data Owner user has access to sensitive information and wants to allow an Application Owner to process this information.
This sensitive data is stored in a file "conditions.csv"

In this example, the file is encrypted, uploaded to a storage solution (AWS s3), and a dataset is configured with credentials and an encryption key for enclave access/processing:

Obtain a copy of the data locally.

wget https://storage.googleapis.com/synthea-public/synthea_sample_data_csv_apr2020.zip
unzip synthea_sample_data_csv_apr2020.zip

To encrypt the file “conditions.csv”, generate a key locally, encrypt the file and store the key in a KMS securely.
1. Generate an encryption key.
```
 xxd -u -l 32 -p /dev/random | tr -d '\n' > ./key.hex
```
2. Encrypt the file.
```
./aes_256_gcm.py enc -K ./key.hex -in ./csv/conditions.csv -out ./conditions.csv.enc
```
  The encrypt utility is provided with examples in the Rust-SGX repo. It is also available here.

Upload the encrypted file to a secure storage location such as AWS S3.

aws s3 --profile upload cp ./conditions.csv.enc s3://fortanix-pocs-data/conditions.csv.enc

Generate a pre-signed URL to access the file. This is done to avoid inserting the whole AWS SDK in the example:
```
aws s3 presign --profile download s3://fortanix-pocs-data/conditions.csv.enc --expires-in 86400
https://fortanix-pocs-data.s3.amazonaws.com/conditions.csv.enc?AWSAccessKeyId=AKIAXO5N6GGNCNV353WS&Signature=PcpH99nszG2%2Fv85z4IbgwgVDywc%3D&Expires=1613817035
```
The URL above is split into two parts:
- The location: https://fortanix-pocs-data.s3.amazonaws.com/conditions.csv.enc
- Query parameters: AWSAccessKeyId=AKIAXO5N6GGNCNV353WS&Signature=PcpH99nszG2%2Fv85z4IbgwgVDywc%3D&Expires=1613817035 which is base64 encoded into:
  QVdTQWNjZXNzS2V5SWQ9QUtJQVhPNU42R0dOQ05WMzUzV1MmU2lnbmF0dXJlPVBjcEg5OW5zekcyJTJGdjg1ejRJYmd3Z1ZEeXdjJTNEJkV4cGlyZXM9MTYxMzgxNzAzNQ==
At this point, users accessing the URL above with full query string parameters will be able to download the encrypted file.
NOTE
If you access the URL without the string following '?', you will get a 403 forbidden.
Hence, treat the query parameters as access credentials.

Perform the following steps to create an input dataset:

In the CCM UI left navigation panel, click Dataverse → DATASETS, and then click ADD DATASET to create a dataset.
Figure 1: Add dataset
In the Add dataset form:
1. Name – Enter the name of the dataset. For example: Conditions Data.
2. Description (optional) – Enter the description of the dataset. For example: Patients with associated conditions.
3. Labels (optional) – Attach one or more key-value labels to the dataset. For example: Key: Location and Value: East US
4. Group– Select the required group name from the drop down menu to associate this dataset with that group.
5. Location – The AWS S3 URL where data can be accessed. For example, https://fortanix-pocs-data.s3.amazonaws.com/conditions.csv.enc .
6. Long Description (optional) – Enter the content in a GitHub-flavoured Markdown file format. You can also use Fetch Long Description to get the Markdown file content from an external URL. Ensure that Cross-Origin Resource Sharing (CORS) is enabled on the external URL so that Fortanix CCM can access it. For more information, refer to the steps for AWS and Azure.
  The following is the sample description:
```
- Strikethrough Text
~~It is Strikethrough test..~~

- Blockquote Text
> Test Blockquote.

- Bold
**A sample Description.**

- Italic
*It is Italics*

- Bold and Italic
***Bold and Italics text***

- Link
This is [Website](https://www.fortanix.com/)?
```
7. Credentials – the credentials needed to access the data. For example:
```
{
  "query_string": "QVdTQWNjZXNzS2V5SWQ9QUtJQVhPNU42R0dOQ05WMzUzV1MmU2lnbmF0dXJlPVBjcEg5OW5zekcyJTJGdjg1ejRJYmd3Z1ZEeXdjJTNEJkV4cGlyZXM9MTYxMzgxNzAzNQ==",
  "encryption": {
    "key": "63F0E4C07666126226D795027862ACC5848E939881C3CFE8CB3EB47DD7B3D24A"
}
```
  NOTE
  The credentials are only passed as text when creating the dataset over an HTTPS connection.
  It is then stored in a KMS (Fortanix Data Security Manager) and is only accessible to approved enclaves.
  Not even the Data Owner can retrieve the credentials.
Click ADD DATASET to create the input dataset.

5.2 Output User

The result of the processing can also be considered confidential. As such, you want the enclave to first encrypt the data with a key that you will provide and upload it to storage that you control.
Steps to achieve this are as follows:

Generate an encryption key.

xxd -u -l 32 -p /dev/random | tr -d '\n' > ./key.hex

Generate a pre-signed URL for enclaves to upload data. AWS CLI can only create URLs for downloading the file and not for uploading them. To achieve this, use a Python script using boto3.
We use a python script because AWS CLI can currently only generate URLs for download and not for upload.
```
./presign.py upload fortanix-pocs-data conditions_output.csv.enc 86400
https://fortanix-pocs-data.s3.amazonaws.com/conditions_output.csv.enc?AWSAccessKeyId=AKIAXO5N6GGNNMRXVKPA&Signature=HFvhxaiKY0cGR9XqgGLp5zcAWac%3D&Expires=1613817880
```
Where, presign.py code is:
```
!/usr/bin/python3
import boto3
import sys
session = boto3.Session(profile_name=sys.argv[1])
s3_client = session.client('s3')
print(s3_client.generate_presigned_url('put_object', Params={'Bucket':sys.argv[2], 'Key':sys.argv[3]}, ExpiresIn=int(sys.argv[4]), HttpMethod='PUT'))
```
The URL above is split into two parts:
- The location: https://fortanix-pocs-data.s3.amazonaws.com/conditions_output.csv.enc
- Query parameters: AWSAccessKeyId=AKIAXO5N6GGNNMRXVKPA&Signature=HFvhxaiKY0cGR9XqgGLp5zcAWac%3D&Expires=1613817880 which is base64 encoded into:
  QVdTQWNjZXNzS2V5SWQ9QUtJQVhPNU42R0dOTk1SWFZLUEEmU2lnbmF0dXJlPUhGdmh4YWlLWTBjR1I5WHFnR0xwNXpjQVdhYyUzRCZFeHBpcmVzPTE2MTM4MTc4ODA=
Create an output dataset with the following sample values:
1. Name – The dataset name. For example, conditions_output.csv.enc.
2. Description (optional) – The dataset description.
3. Labels (optional)– attach one or more key-value labels to the dataset. For example: Key: Location and Value: East US.
4. Group – Select the required group name from the drop down menu to associate this dataset with that group.
5. Location – The URL where data can be accessed. For example: https://fortanix-pocs-data.s3.amazonaws.com/conditions_output.csv.enc
6. Long Description (optional) – Enter the content in GitHub-flavoured Markdown file format. You can also use Fetch Long Description to get the Markdown file from an external URL. Ensure that Cross-origin resource sharing (CORS) is enabled on the external URL so that Fortanix CCM can access it. For more information, refer to the steps for AWS and Azure.
  The following is the sample long description in Markdown format:
```
- Strikethrough Text
~~It is Strikethrough test..~~

- Blockquote Text
> Test Blockquote.

- Bold
**A sample Description.**

- Italic
*It is Italics*

- Bold and Italic
***Bold and Italics text***

- Link
This is [Website](https://www.fortanix.com/)?
```
7. Credentials – The credentials needed to access the data. For example:
```
{
  "query_string": "QVdTQWNjZXNzS2V5SWQ9QUtJQVhPNU42R0dOTk1SWFZLUEEmU2lnbmF0dXJlPUhGdmh4YWlLWTBjR1I5WHFnR0xwNXpjQVdhYyUzRCZFeHBpcmVzPTE2MTM4MTc4ODA=","encryption": {
    "key": "63F0E4C07666126226D795027862ACC5848E939881C3CFE8CB3EB47DD7B3D24A"
  }
}
```

6.0 Create Application Configuration

For this specific EDP application, there are no key or value pairs to specify, but you must specify the input and output ports.

You must also specify how the application connects to a dataset using input and output ports in an app configuration.

Perform the following steps to create an application configuration:

In the Fortanix CCM UI left navigation panel, navigate to Applications → CONFIGURATIONS.
Figure 2: Create application configuration
Click ADD CONFIGURATION to add a new configuration.
1. Configuration Name – Enter a name for the configuration. For example: PatientsConfig
2. Description: Enter a description for the configuration.
3. Group: Select the required group name from the drop down menu.
4. Build: Select the application build for which the configuration will be created.
5. Ports: Specify the ports to be used in the workflow. Multiple ports can be added, depending on the required connections. For example:input,output,heartbeat, and so on.
6. Label Details (optional): Add one or more key-value labels to the configuration.
7. Configuration items: Click ADD CONFIGURATION ITEM and define key-value pairs used to configure the application.
NOTE
For ACI applications, Fortanix permits only files in the path /opt/fortanix/.
Click ADD CONFIGURATION to save the configuration.

7.0 Create a Workflow

Perform the following steps to create a Workflow:

In the CCM UI left navigation panel, click Workflows.
On the Workflows page, click ADD WORKFLOW to create a new workflow.
Figure 3: Create a workflow
In the Add Workflow form:
1. Name: Enter a name for the workflow.
2. Group: Select the Fortanix Armor IAM group for the workflow. If you do not select a group, Fortanix CCM uses the default group.
3. Click ADD WORKFLOW to create the workflow.
Add an application to the workflow graph. Drag the Application icon from the component panel and drop it into the graph area.
Click ADD APPLICATION. In the Add Application form, select an existing application name and image. Select a build for this application. For example, <my-registry>/simple-python-sgx:latest.
Where, <my-registry> is the location of your container registry.
Select an existing app configuration or create a new app configuration using ADD NEW CONFIGURATION for the application build.
Add input and output datasets to the workflow graph. Drag the Dataset icon into the graph area and click ADD DATASET.
In the DATASET form, select an existing dataset, or click ADD DATASET to create a new dataset.
Connect the workflow components. Establish connections between datasets and applications as follows:
1. Connect the Input Dataset to the Application using the Input target port.
2. Connect the Application to the Output Dataset using the Output target port.
After the workflow is completed, click SAVE AND REQUEST APPROVAL to initiate the workflow approval process.
The workflow remains in a pending state until it receives approval from all users. In the Pending menu item, click VIEW REQUEST to approve a Workflow.
In the APPROVAL REQUEST FOR CREATING WORKFLOW dialog, you can either APPROVE or DECLINE a workflow.
WARNING
When a draft Workflow is submitted for approval, it will be removed from the drafts list, and editing it directly will no longer be possible once it is in a "pending" or "approved" state.
NOTE
- A user can also approve/decline a Workflow from the CCM Tasks menu item.
- Notice that the users who have approved the Workflow have a green tick against their icon.
All the users of a Workflow must approve the Workflow to finalize it. If a user declines a Workflow, the Workflow is rejected. When all the users approve the Workflow, it is deployed.
1. CCM configures apps to access the datasets.
2. CCM creates the Workflow Application Configs.
3. CCM returns the list of hashes needed to start the apps.

8.0 Run the Application

After a Workflow is approved by all the users, the Applications will have the Workflow Application Configurations provided to them. This configuration has information such as which Datasets or Apps they are connected to, any user-provided files or values to be provided within an enclave, and so on.

We provide a configuration to applications using an identifier passed as an input argument.

This identifier is a sha256sum of items that you need to secure from the configuration and workflow.

Fortanix CCM will also embed this identifier inside the certificates it issues so that it is clear what configuration is used for the KMS to allow access to credentials.

It embeds this inside a subject alternate name:

<identifier>.<mrenclave>.id.fortanix.cloud

With the identifier above, the KMS that stores the dataset credentials will authenticate and give credentials only to applications that present a proper certificate. When the application starts, CCM will keep track of which applications are allowed to access which configurations using the identifier.

Perform the following steps to view the Application Identifier:

Click the application in the approved Workflow graph.
In the detailed view of the Workflow application, copy the value of the Runtime configuration hash. This ID is used to run the application.

To run the application-

Without Docker, execute the following command:

$ ./ftxsgx-runner ./harmonize.sgxs d2a2b86fa7d0afefa4a906292a09e42d56eeaf014a3c549b3320cefada90aab9

Download finished from location: https://fortanix-pocs-data.s3.amazonaws.com/conditions.csv.enc
Upload finished at location: https://fortanix-pocs-data.s3.amazonaws.com/conditions_output.csv.enc

With Docker, execute the following command:

docker run --privileged -d -v /var/run/aesmd:/var/run/aesmd --volume /dev:/dev --net="host" -it patients-research:1.0 d2a2b86fa7d0afefa4a906292a09e42d56eeaf014a3c549b3320cefada90aab9

Download finished from location: https://fortanix-pocs-data.s3.amazonaws.com/conditions.csv.enc
Upload finished at location: https://fortanix-pocs-data.s3.amazonaws.com/conditions_output.csv.enc

The Data Output Owner can view the output using the following steps:

Download the output file:

aws s3 --profile download cp s3://fortanix-pocs-data/conditions_output.csv.enc .
download: s3://fortanix-pocs-data/conditions_output.csv.enc to ./conditions_output.csv.enc

The encrypted file contents are as follows:

$ hexdump -C ./conditions_output.csv.enc 

00000000 01 e2 c1 f6 f5 6c d2 d5 2f 05 83 72 1c 83 91 94 |.....l../..r....|
00000010 92 1b c4 fc 98 30 66 a4 40 33 ba 63 de 5f 10 eb |.....0f.@3.c._..|
00000020 c8 d5 22 48 de ae 4b 34 61 07 11 b7 5a f5 b2 0d |.."H..K4a...Z...|
00000030 29 b5 49 00 6c d7 e7 62 76 f6 a5 74 c7 0b 5b 7b |).I.l..bv..t..[{|
00000040 4d 52 93 dc 92 a4 95 53 a1 ac c6 09 d0 22 33 2f |MR.....S....."3/|
00000050 56 45 c8 25 22 64 b3 07 78 f1 e6 54 6f a0 43 34 |VE.%"d..x..To.C4|
00000060 e4 10 02 21 aa f8 98 c2 4a b3 0e 5f 4c 99 b5 75 |...!....J.._L..u|
00000070 34 1d 60 bf 00 1d cc 91 0d a7 08 f4 c8 55 42 58 |4.`..........UBX|
00000080 da 00 3d 5c 02 1e 86 5e 17 8c b6 45 2a 76 35 70 |..=\...^...E*v5p|
00000090 6b 7e 84 dc 53 4a f1 8a e2 01 04 aa b9 43 75 c0 |k~..SJ.......Cu.|
000000a0 fd f4 55 2d 16 71 8e dd dd 18 25 68 03 93 e7 64 |..U-.q....%h...d|
000000b0 60 d3 00 6f 98 22 a3 92 a7 2a f0 8a 79 ff bf e3 |`..o."...*..y...|
000000c0 4d 24 0d 8e f7 3e bf 50 bf 1a 52 9d b8 bd a0 b2 |M$...>.P..R.....|
000000d0 14 87 f5 c0 30 43 8c ae b9 f6 de 54 23 19 f0 2c |....0C.....T*..,|
000000e0 f3 e7 b8 9c 59 03 30 87 db 7a cf f0 82 3f 7f 4b |....Y.0..z...?.K|
000000f0 d9 10 c2 1b ee 64 c1 28 6d 38 70 f1 7e 6b 90 1f |.....d.(m8p.~k..|
00000100 f5 c1 17 3e db 1d 57 6a 96 27 fe e2 71 54 dc 3d |...>..Wj.'..qT.=|
00000110 36 a7 bc 4e 07 05 7a cb f9 af e9 bc 01 87 d4 8c |6..N..z.........|
00000120 3a c3 f4 d1 1b |:....|
00000125

Decrypt the file:

 ./aes_256_gcm.py dec -K ./key.hex -in ./conditions_output.csv.enc -out ./conditions_output.csv

The file now has the expected output:

cat conditions_output.csv
count                               8376 
unique                               129 
top           Viral sinusitis (disorder) 
freq                                1248 
Name: DESCRIPTION, dtype: object

When the App Owner starts the application with the application config identifier:

Applications will request an attestation certificate from the NodeAgent with the identifier as part of the report data.
The application requests an application certificate from the NodeAgent.
The CCM verifies that the application is allowed to access the configuration.
The application requests from CCM its configuration by providing its certificate provisioned above as an authentication mechanism.
The CCM does certificate authentication, extracts the application identifier from the certificate, and sends back the configuration corresponding to that identifier.
The application verifies and applies the configuration hash.
The application gets the credentials from URLs in the config.
The application authenticates and reads/writes data from the datasets.