Workflow Applications Using Fortanix EDP

1.0 Introduction

Workflow graphs are maps that show how generic applications are connected to datasets and other generic applications. These are collaborative objects where multiple users can provide their own objects and approvals. For more details refer to the User Guide: Workflows.

This article describes how to create and deploy Fortanix CCM Workflows using a simple data processing example.

1.1 A Research Portal for Patient Data

This example demonstrates a Fortanix CCM workflow with rust applications deployed securely using Fortanix EDP.

The Workflow provides stratification of patient data by demographic attributes (for example: age, gender, ethnicity).

For input data,  a synthetic file containing the medical history of fictive patients is used.

The data used for this example is available from https://synthea.mitre.org/downloads

File: 1K Sample Synthetic Patient Records, CSV[mirror]: 9 MB

This is made available via:
Jason Walonoski, Mark Kramer, Joseph Nichols, Andre Quina, Chris Moesel, Dylan Hall, 
Carlton Duffett, Kudakwashe Dube, Thomas Gallagher, Scott McLachlan, 
Synthea: An approach, method, and software mechanism for generating synthetic patients 
and the synthetic electronic health care record, Journal of the American Medical 
Informatics Association, Volume 25, Issue 3, March 2018, 
Pages 230–238, https://doi.org/10.1093/jamia/ocx079

The input and output dataset represents patient information collected by a different healthcare provider and contains the following elements

• Patient demographic information
• Patient conditions
• Patient encounters
• Patient medications

2.0 Create CCM Workflow

2.1 Sign up and Log in

1. Sign in to Fortanix CCM using the URL: https://ccm.fortanix.com/.
   Figure 1: Sign up and Log in
   For more details, refer to the https://support.fortanix.com/hc/en-us/articles/360034373551-User-s-Guide-Logging-in#SignUp.

2.2 Get all Accounts

To get the accounts using Fortanix CCM UI, log in to Fortanix CCM to see the available accounts.

Figure 2: Get All Accounts

For more details refer to https://support.fortanix.com/hc/en-us/articles/360034373551-User-s-Guide-Logging-in#LogIn.

2.3 Select the Account

To select an account using Fortanix CCM UI, click SELECT to enter the account.

Figure 3: Select the Account

2.4 Invite Users to Join CCM

An Administrator invites users who are Data Owners, App Owners, and Output Owners to join a CCM account and create a workflow graph.

To invite users, click the Users icon  and in the Users page, click INVITE USER to invite new users to the account.

Figure 4: Invite Users

For more details refer to https://support.fortanix.com/hc/en-us/articles/360043093352-User-s-Guide-Invite-Users.

The new users (Data Owners and App Owners) must join the CCM account and create Datasets and App Configs as described in the following sections.

2.5 Writing the Application

Fortanix EDP applications are rust applications using rust-sgx SDK. 

For the EDP application, the user needs to write the application to perform operations such as download and decrypt the patient’s input conditions data, process it, and encrypt and upload the output conditions data to a secure AWS S3 location.

The functionality for this is available in the Fortanix repository: <URL TBD>. These operations are done using a Python script for an Enclave OS application. 

fn process(decrypted: Vec<u8>) -> Result<String, String> {
    [allow(non_snake_case)]
    [derive(Debug, Deserialize)]
    struct Record {
        START: String,
        STOP: String,
        PATIENT: String,
        ENCOUNTER: String,
        CODE: String,
        DESCRIPTION: String,
    };
    let mut statistics = HashMap::<String, u32>::new();
    let mut count : u32 = 0;
    let mut rdr = csv::Reader::from_reader(&*decrypted);
    for i in rdr.deserialize() {
        let record: Record = i.map_err(|_| "Invalid CSV data".to_string())?;
        statistics.entry(record.DESCRIPTION).and_modify(|e| *e += 1).or_insert(1);
        count += 1;
    }
   
    let last_entry = statistics.iter().max_by(|a, b| a.1.cmp(&b.1)).ok_or("No entries in CSV")?;
    let top = last_entry.0;
    let freq = last_entry.1;
    let unique = statistics.len();
   
    let result = format!("count {:<50}\nunique {:<50}\ntop {:<50}\nfreq {:<50}\nName: DESCRIPTION, dtype: object\n",
    count, unique, top, freq);
   
    Ok(result)
}

3.0 Sign the Application and Create Docker Image

Next, the user must sign the Application. In this example we use a development certificate using the following script:

cargo build --target=x86_64-fortanix-unknown-sgx
app_bin=./target/x86_64-fortanix-unknown-sgx/debug/harmonize

signing_cert=$(dirname ${app_bin})/private.pem
openssl genrsa -3 3072 > ${signing_cert}

ftxsgx-elf2sgxs ${app_bin} --heap-size 10240000 --stack-size 10240000 --threads 5 --debug
sgxs-sign ${app_bin}.sgxs "${app_bin}.sigstruct" --key ${signing_cert} -p 1 -v 1

You can also build a docker image using the following file. This file copies harmonize.sgxs/ftxsgx-runner from the current folder:

FROM ubuntu:20.04
RUN apt-get update && apt-get install -y --no-install-recommends libssl-dev
WORKDIR /root/
COPY ./harmonize.sgxs .
COPY ./ftxsgx-runner .
ENTRYPOINT ["/root/ftxsgx-runner", "/root/harmonize.sgxs"]

The parameters for registering the app are in the file ./target/x86_64-fortanix-unknown-sgx/debug/harmonize.sigstruct created by the sign operation.

4.0 Create an EDP Application Using UI

1. To create an EDP application using Fortanix CCM UI, click the Applications icon , and in the Applications page click +APPLICATION to create an application.
   Figure 5: Create an EDP Application
2. Create an EDP Application.
   
   Figure 6: Add an EDP Application
   For more details refer to https://support.fortanix.com/hc/en-us/articles/360043527431-User-s-Guide-Add-and-Edit-an-Application#AddEDPApplication.
3. Create an image of the application.
   
   Figure 7: Add Image
   For more details refer to https://support.fortanix.com/hc/en-us/articles/360043529411-User-s-Guide-Create-an-Image#CreateanImageforEDPApplications.
4. Fetch the build whitelisting tasks using the Fortanix CCM Tasks tab.
   
   Figure 8: Fetch Tasks for Approval 
5. Approve the task.
   
   Figure 9: Approve the Task

5.0 Create Input and Output Datasets

As input and output datasets to this workflow, two synthetic datasets are used that contain the medical history of synthetic patients.

In this example, create the following data points and credentials for download/upload:

• AWS S3 URL for Input dataset:
  https://fortanix-pocs-data.s3.amazonaws.com/conditions.csv.enc
• AWS S3 URL for Output dataset:
  https://fortanix-pocs-data.s3.amazonaws.com/conditions_output.csv.enc
• Associated encryption keys and access credentials: ./key.hex

5.1 Input User

Consider that the Data Owner user has access to sensitive information and wants to allow an Application Owner to process this information.
This sensitive data is stored in a file "conditions.csv"

In this example, the file is encrypted, uploaded to a storage solution (AWS s3), and a dataset is configured with credentials and an encryption key for enclave access/processing:

1. Obtain a copy of the data locally.

   wget https://storage.googleapis.com/synthea-public/synthea_sample_data_csv_apr2020.zip
   unzip synthea_sample_data_csv_apr2020.zip

2. To encrypt the file “conditions.csv”, generate a key locally, encrypt the file and store the key in a KMS securely.
   1. Generate an encryption key.

       xxd -u -l 32 -p /dev/random | tr -d '\n' > ./key.hex

   2. Encrypt the file.

      ./aes_256_gcm.py enc -K ./key.hex -in ./csv/conditions.csv -out ./conditions.csv.enc

      The encrypt utility is provided with examples in rust-sgx repo. It's also available here.
3. Upload the encrypted file to a secure storage location such as AWS S3. 
   

   aws s3 --profile upload cp ./conditions.csv.enc s3://fortanix-pocs-data/conditions.csv.enc

4. Generate a pre-signed URL to access the file. This is done to avoid inserting the whole AWS SDK in the example:  

   aws s3 presign --profile download s3://fortanix-pocs-data/conditions.csv.enc --expires-in 86400
   https://fortanix-pocs-data.s3.amazonaws.com/conditions.csv.enc?AWSAccessKeyId=AKIAXO5N6GGNCNV353WS&Signature=PcpH99nszG2%2Fv85z4IbgwgVDywc%3D&Expires=1613817035

   The URL above is split into two parts:
   • The location: https://fortanix-pocs-data.s3.amazonaws.com/conditions.csv.enc
   • Query parameters: AWSAccessKeyId=AKIAXO5N6GGNCNV353WS&Signature=PcpH99nszG2%2Fv85z4IbgwgVDywc%3D&Expires=1613817035 which is base64 encoded into:
     QVdTQWNjZXNzS2V5SWQ9QUtJQVhPNU42R0dOQ05WMzUzV1MmU2lnbmF0dXJlPVBjcEg5OW5zekcyJTJGdjg1ejRJYmd3Z1ZEeXdjJTNEJkV4cGlyZXM9MTYxMzgxNzAzNQ==
      
   At this point, users accessing the URL above with full query string parameters will be able to download the encrypted file. 
   
   NOTE

   If you access the URL without the string following '?', you will get a 403 forbidden.
   Hence, treat the query parameters as access credentials.

To create an input dataset:

1. Click the Datasets icon in the CCM left panel and click CREATE DATASET to create a new dataset.
   Figure 10: Create Datasets
2. In the Create new dataset form, enter the following details:
   • Name – the dataset name. For example: Conditions Data.
   • Description (optional) – the dataset description. For example: Patients with associated conditions.
   • Labels (optional)– attach one or more key-value labels to the dataset. For example: Key: Location and Value: East US.
   • Location – the AWS S3 URL where data can be accessed. For example: https://fortanix-pocs-data.s3.amazonaws.com/conditions.csv.enc
   • Long Description (optional)– enter the content in GitHub-flavoured Markdown file format. You can also use the Fetch Long Description button to get the Markdown file from an external URL.
     
     Figure 11: Fetch Long Description Dialog Box
     The following is the sample description:

     - Strikethrough Text
     ~~It is Strikethrough test..~~
     
     - Blockquote Text
     > Test Blockquote.
     
     - Bold
     **A sample Description.**
     
     - Italic
     *It is Italics*
     
     - Bold and Italic
     ***Bold and Italics text***
     
     - Link
     This is [Website](https://www.fortanix.com/)?

   • Credentials – the credentials needed to access the data. For example:

     {
       "query_string": "QVdTQWNjZXNzS2V5SWQ9QUtJQVhPNU42R0dOQ05WMzUzV1MmU2lnbmF0dXJlPVBjcEg5OW5zekcyJTJGdjg1ejRJYmd3Z1ZEeXdjJTNEJkV4cGlyZXM9MTYxMzgxNzAzNQ==",
       "encryption": {
         "key": "63F0E4C07666126226D795027862ACC5848E939881C3CFE8CB3EB47DD7B3D24A"
     }

     NOTE

     • The credentials are only passed as text when creating the dataset over an HTTPS connection.
     • It is then stored in a KMS (Fortanix Data Security Manager) and only accessible to approved enclaves.
     • Not even the Data Owner can retrieve the credentials.
   
   Figure 12: Create Input Dataset
3. Click CREATE DATASET to create the input dataset.

5.2 Output User

The result of the processing can also be considered confidential. As such, you want the enclave to first encrypt the data with a key that you will provide and upload it to a storage that you control.
Steps to achieve this are as follows:

1. Generate an encryption key.

   xxd -u -l 32 -p /dev/random | tr -d '\n' > ./key.hex

2. Generate a pre-signed URL for enclaves to upload data. AWS CLI can only create URLs for downloading the file and not for uploading them. To achieve this, use a Python script using boto3.
   We use a python script because AWS CLI can currently only generate URLs for download and not for upload.

   ./presign.py upload fortanix-pocs-data conditions_output.csv.enc 86400
   https://fortanix-pocs-data.s3.amazonaws.com/conditions_output.csv.enc?AWSAccessKeyId=AKIAXO5N6GGNNMRXVKPA&Signature=HFvhxaiKY0cGR9XqgGLp5zcAWac%3D&Expires=1613817880

   Where, presign.py code is:

   !/usr/bin/python3
   import boto3
   import sys
   session = boto3.Session(profile_name=sys.argv[1])
   s3_client = session.client('s3')
   print(s3_client.generate_presigned_url('put_object', Params={'Bucket':sys.argv[2], 'Key':sys.argv[3]}, ExpiresIn=int(sys.argv[4]), HttpMethod='PUT'))

     The URL above is split into two parts:
   • The location: https://fortanix-pocs-data.s3.amazonaws.com/conditions_output.csv.enc
   • Query parameters: AWSAccessKeyId=AKIAXO5N6GGNNMRXVKPA&Signature=HFvhxaiKY0cGR9XqgGLp5zcAWac%3D&Expires=1613817880 which is base64 encoded into:
     QVdTQWNjZXNzS2V5SWQ9QUtJQVhPNU42R0dOTk1SWFZLUEEmU2lnbmF0dXJlPUhGdmh4YWlLWTBjR1I5WHFnR0xwNXpjQVdhYyUzRCZFeHBpcmVzPTE2MTM4MTc4ODA=
3. Create an output dataset with the following sample values:
   • Name – the dataset name. For example: conditions_output.csv.enc.
   • Description (optional) – the dataset description.
   • Labels (optional)– attach one or more key-value labels to the dataset. For example: Key: Location and Value: East US
   • Location – the URL where data can be accessed. For example: https://fortanix-pocs-data.s3.amazonaws.com/conditions_output.csv.enc
   • Long Description (optional)– enter the content in GitHub-flavoured Markdown file format. You can also use the Fetch Long Description button to get the Markdown file from an external URL.
     
     Figure 13: Fetch Long Description Dialog Box
     The following is the sample long description in Markdown format:

     - Strikethrough Text
     ~~It is Strikethrough test..~~
     
     - Blockquote Text
     > Test Blockquote.
     
     - Bold
     **A sample Description.**
     
     - Italic
     *It is Italics*
     
     - Bold and Italic
     ***Bold and Italics text***
     
     - Link
     This is [Website](https://www.fortanix.com/)?

   • Credentials – the credentials needed to access the data. For example:

     {
       "query_string": "QVdTQWNjZXNzS2V5SWQ9QUtJQVhPNU42R0dOTk1SWFZLUEEmU2lnbmF0dXJlPUhGdmh4YWlLWTBjR1I5WHFnR0xwNXpjQVdhYyUzRCZFeHBpcmVzPTE2MTM4MTc4ODA=",
       "encryption": {
         "key": "63F0E4C07666126226D795027862ACC5848E939881C3CFE8CB3EB47DD7B3D24A"
       }
     }

      
   
   Figure 14: Create Output Dataset

6.0 Create Application Config

For this specific EDP application, there are no config parameters to specify. (other applications can choose to have custom behavior based on parameters).

With that said, you still need to specify how the application connects to a dataset.
That means specifying input and output ports in an App Configuration. Steps to do that are as follows:

1. Click the Applications icon in the CCM left panel and from the left menu select Configurations.
2. Click ADD CONFIGURATION to add a new configuration.
   Figure 13: Create App Configuration
   1. Image – select the “patients-Research:1.0” application image for which you want to create a configuration.
   2. Name and Description – Enter a name and description of the configuration. For example: PatientsConfig.
   3. Ports – Enter the connections to be used in the workflow. You can add multiple ports depending on how the connection should work. For example: “input”, “output”, “heartbeat”, and so on.
   4. Labels – attach one or more key-value labels to the app config.
   5. Configuration items – These are key-value pairs used for configuring the app. In the ADD APPLICATION CONFIGURATION window, fill the following:
   
   
   Figure 16: Save Configuration
3. Click SAVE CONFIGURATION to save the configuration.
   Figure 17: Configuration Saved

7.0 Create a Workflow

To create a workflow:

1. Click the Workflows icon in the CCM left panel.
2. In the Workflows page, click +WORKFLOW to create a new workflow.
   Figure 18: Create Workflow
3. In the CREATE WORKFLOW dialog, enter the Workflow Name and Description (optional). Click CREATE to go to the Workflow graph.
4. Add an app to the Workflow graph. To add an app to a Workflow graph, drag the App icon and drop it into the graph area. Click +APPLICATION. In the ADD APPLICATION dialog, the App Owner must select an existing application image, for example: Patients-Research:1.0 from the list of available application images.
5. For the selected application image, the App Owner must create an app config/add an existing app config.
6. Click SELECT APPLICATION to select the application.
   Figure 19: Add App to Workflow
7. Click SAVE AS DRAFT to save the draft.
   Figure 20: Save Workflow Draft
8. To access the draft Workflow, click the Draft tab in the Workflows left menu.
9. Add an input and output dataset to the Workflow graph. To add a dataset to a Workflow graph, drag the dataset icon and drop it into the graph area. Click +DATASET. In the DATASET dialog, the Data Owner must select from existing datasets that were created in the section above.
   Figure 21: Add Input Dataset to Workflow
   Figure 22: Add Output Dataset to Workflow
10. Create connections between the applications and input/output datasets. To do that, drag the Input Dataset connection point and join it to the Application connection point. In the SELECT PORTS window, select the Target Port as “input”. Repeat the same to connect the Application to the Output Dataset and select the Target Port as “Output”.
    Figure 23: Create Connection
11. If the Workflow is complete, the user clicks the REQUEST APPROVAL button to generate the approval process for the Workflow.
    Figure 24: Request Workflow Approval
    WARNING

    When a draft Workflow is submitted for approval, it will be removed from the drafts list and the user will no longer be able to edit it directly once it is in “pending” or "approved" state.
12. The workflow is in “pending” state until all the users approve it. In the Pending tab click SHOW APPROVAL REQUEST to approve a Workflow.
    Figure 25: Workflow Pending Approval
13. In the APPROVAL REQUEST - CREATE WORKFLOW dialog, click APPROVE to approve the workflow or DECLINE to reject a workflow.
    Figure 26: Approve the Workflow
    NOTE

    • A user can also approve/decline a Workflow from the CCM Tasks tab.
    • Notice that the users who have approved the workflow have a green tick against their icon.
14. All the users of a Workflow must approve the Workflow to finalize it. If a user declines a Workflow, the Workflow is rejected. When all the users approve the Workflow, it is deployed.
    1. CCM configures apps to access the datasets.
    2. CCM creates the Workflow Application Configs.
    3. CCM returns the list of hashes needed to start the apps.

8.0 Run the Application

After a Workflow is approved by all the users, the Applications will have the Workflow Application Configurations provided to them. This configuration has information such as which Datasets or Apps they are connected to, any user-provided files or values to be provided within an enclave, and so on.

We provide a configuration to applications using an identifier passed as an input argument.

This identifier is a sha256sum of items that you need to secure from the configuration and workflow. 

Fortanix CCM will also embed this identifier inside the certificates it issues so that it is clear what configuration is used for the KMS to allow access to credentials.

It embeds this inside a subject alternate name:

<identifier>.<mrenclave>.id.fortanix.cloud

With the identifier above, the KMS that stores the dataset credentials will authenticate and give credentials only to applications that present a proper certificate. When the application starts, CCM will keep track of which applications are allowed to access which configurations using the identifier.

To view the Application Identifier:

1. Click the application in the approved Workflow graph.
   Figure 27: Workflow Application Detailed View
2. In the detailed view of the Workflow application, copy the value of Runtime configuration hash. This ID is used to run the application.
   Figure 28: Copy the App Identifier
3. To run the application-
   1. Without docker, execute the following command:

      $ ./ftxsgx-runner ./harmonize.sgxs d2a2b86fa7d0afefa4a906292a09e42d56eeaf014a3c549b3320cefada90aab9

      Download finished from location: https://fortanix-pocs-data.s3.amazonaws.com/conditions.csv.enc
      Upload finished at location: https://fortanix-pocs-data.s3.amazonaws.com/conditions_output.csv.enc

   2. With docker, execute the following command:

      docker run --privileged -d -v /var/run/aesmd:/var/run/aesmd --volume /dev:/dev --net="host" -it patients-research:1.0 d2a2b86fa7d0afefa4a906292a09e42d56eeaf014a3c549b3320cefada90aab9

      Download finished from location: https://fortanix-pocs-data.s3.amazonaws.com/conditions.csv.enc
      Upload finished at location: https://fortanix-pocs-data.s3.amazonaws.com/conditions_output.csv.enc

4. The Data Output Owner can view the output using the following steps:
   1. Download the output file:

      aws s3 --profile download cp s3://fortanix-pocs-data/conditions_output.csv.enc .
      download: s3://fortanix-pocs-data/conditions_output.csv.enc to ./conditions_output.csv.enc

   2. The encrypted file contents are as below:

      $ hexdump -C ./conditions_output.csv.enc 
      
      00000000 01 e2 c1 f6 f5 6c d2 d5 2f 05 83 72 1c 83 91 94 |.....l../..r....|
      00000010 92 1b c4 fc 98 30 66 a4 40 33 ba 63 de 5f 10 eb |.....0f.@3.c._..|
      00000020 c8 d5 22 48 de ae 4b 34 61 07 11 b7 5a f5 b2 0d |.."H..K4a...Z...|
      00000030 29 b5 49 00 6c d7 e7 62 76 f6 a5 74 c7 0b 5b 7b |).I.l..bv..t..[{|
      00000040 4d 52 93 dc 92 a4 95 53 a1 ac c6 09 d0 22 33 2f |MR.....S....."3/|
      00000050 56 45 c8 25 22 64 b3 07 78 f1 e6 54 6f a0 43 34 |VE.%"d..x..To.C4|
      00000060 e4 10 02 21 aa f8 98 c2 4a b3 0e 5f 4c 99 b5 75 |...!....J.._L..u|
      00000070 34 1d 60 bf 00 1d cc 91 0d a7 08 f4 c8 55 42 58 |4.`..........UBX|
      00000080 da 00 3d 5c 02 1e 86 5e 17 8c b6 45 2a 76 35 70 |..=\...^...E*v5p|
      00000090 6b 7e 84 dc 53 4a f1 8a e2 01 04 aa b9 43 75 c0 |k~..SJ.......Cu.|
      000000a0 fd f4 55 2d 16 71 8e dd dd 18 25 68 03 93 e7 64 |..U-.q....%h...d|
      000000b0 60 d3 00 6f 98 22 a3 92 a7 2a f0 8a 79 ff bf e3 |`..o."...*..y...|
      000000c0 4d 24 0d 8e f7 3e bf 50 bf 1a 52 9d b8 bd a0 b2 |M$...>.P..R.....|
      000000d0 14 87 f5 c0 30 43 8c ae b9 f6 de 54 23 19 f0 2c |....0C.....T*..,|
      000000e0 f3 e7 b8 9c 59 03 30 87 db 7a cf f0 82 3f 7f 4b |....Y.0..z...?.K|
      000000f0 d9 10 c2 1b ee 64 c1 28 6d 38 70 f1 7e 6b 90 1f |.....d.(m8p.~k..|
      00000100 f5 c1 17 3e db 1d 57 6a 96 27 fe e2 71 54 dc 3d |...>..Wj.'..qT.=|
      00000110 36 a7 bc 4e 07 05 7a cb f9 af e9 bc 01 87 d4 8c |6..N..z.........|
      00000120 3a c3 f4 d1 1b |:....|
      00000125

   3. Decrypt the file:

       ./aes_256_gcm.py dec -K ./key.hex -in ./conditions_output.csv.enc -out ./conditions_output.csv

   4. The file now has the expected output:

      cat conditions_output.csv
      count                               8376 
      unique                               129 
      top           Viral sinusitis (disorder) 
      freq                                1248 
      Name: DESCRIPTION, dtype: object

When the App Owner starts the application with the application config identifier:

1. Applications will request an attestation certificate from the NodeAgent with the identifier as part of the report data.
2. The application requests an application certificate from the NodeAgent.
3. The CCM verifies that the application is allowed to access the configuration.
4. The application requests from CCM its configuration by providing its certificate provisioned above as an authentication mechanism.
5. The CCM does certificate authentication, extracts the application identifier from the certificate, and sends back the configuration corresponding to that identifier.
6. The application verifies and applies the configuration hash.
7. The application gets the credentials from URLs in the config.
8. The application authenticates and reads/writes data from the datasets.