DSM Accelerator Webservice - External Logging

1.0 Introduction

This article describes the different procedures tested by Fortanix to export Docker container logs for Data Security Manager (DSM) Accelerator Webservice to a remote Security Information and Event Management (SIEM) server, such as Splunk, Syslog, Datadog, Elasticsearch, or Sumo Logic.

The following solutions are described in this article:

Exporting Fortanix DSM Accelerator Webservice container logs to an Elasticsearch Extract, Transform, and Load (ETL) tool such as Logstash.
Exporting Fortanix DSM Accelerator Webservice container logs to a Syslog server.
Exporting Fortanix DSM Accelerator Webservice container logs to a Syslog server with fluentd daemon.

NOTE
The procedures described in this article only serve as examples for the Fortanix DSM Accelerator Webservice with a logging setup. Production deployments for these solutions would require appropriate DevOps or infrastructure support.

2.0 Export Logs to Logstash

2.1 Deployment Type

This procedure involves running the Fortanix DSM Accelerator Webservice container application in a Docker system.

2.1 Setup

In this procedure, the Fortanix DSM Accelerator Webservice container logs will be produced at the location: /var/lib/docker/containers/<container_id>

Perform the following steps:

To keep the log rotation with the maximum number of files and specify a maximum size for each file, a Docker configuration file can be created.
The following is an example configuration file:
```
/etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "labels": "dsma_diag",
    "env": "os,customer"
  }
}
```
The above configuration file specifies a log rotation of 3 files, each with a 10 MB size.
Run the following command to restart the Docker daemon so that the configuration file created in Step 1 comes into effect:
```
$ sudo systemctl restart docker
```
Alternatively, these configuration settings can also be provided when you run the Docker container using the following command:
```
sudo docker run -it --network host --log-driver json-file --log-opt max-size=10m --log-opt max-file=3
```
NOTE
- The above setup can be coupled with a log scraper like Filebeat to forward the container logs to Logstash for instance which is an ETL tool, to transform data using data pipelines.
- The configuration file settings will apply by default to all containers running with that Docker daemon on a host. A container can also be provided with the arguments while running, as described in the alternate way above, to override the default settings.

The following table lists the advantages and disadvantages of this procedure:

SL. NO.	ADVANTAGES	DISADVANTAGES
1	This procedure involves a simple Docker configuration with log rotation and limiting the maximum log size.	Since the Docker container is agnostic towards any scrapper or log collector, this procedure does not require any configuration on the Docker end to allow log scrapping.
2	This procedure requires the logs to be scrapped periodically in order to retain all logging information.	Some log scrapers like Filebeat can only forward logs to an Elasticsearch, Logstash, and Kibana (ELK) stack.

3.0 Export Logs to Syslog Server

3.1 Deployment Type

This procedure involves running the Fortanix DSM Accelerator Webservice container application in a Docker system that is configured to forward logs to a Syslog server directly.

3.2 Setup

This procedure simulates a remote syslog server using rsyslog as the server mode. The Fortanix DSM Accelerator Webservice container logs will be directed to the remote Syslog server instead of being produced at the location: /var/lib/docker/containers/<container_id> so as to prevent disk space exhaustion.

Perform the following steps:

Configure the Syslog driver in Docker using the configuration file: etc/docker/daemon.json:
```
{
  "log-driver": "syslog",
   "log-opts": {
    "syslog-address": "tcp://host_ip:514",
    "syslog-format": "rfc5424"
  }
}
```
Where, syslong-address can be the IP address or the fully qualified domain name (FQDN) of the Syslog server.
Alternatively, these configuration settings can also be provided when you run the Docker container using the following command:
```
sudo docker run -it --network host --log-driver syslog --log-opt syslog-address=tcp://host_ip:514 --log-opt syslog-format=rfc5424
```
NOTE
The configuration file settings will apply by default to all containers running with that Docker daemon on a host. A container can also be provided the arguments while running as described in the alternate way above to override the default settings.

With the above setup, when the Docker container is running, the logs will be forwarded to the Syslog server, and this can be observed by tailing the logs on the server side using the following command:

sudo tail -f /var/log/syslog

Sample output:

Jun 15 15:58:28 linux-adm bbc5eb374230[115923] INFO dsma::server - Initialized DsmAServer
Jun 15 15:58:29 linux-adm bbc5eb374230[115923] INFO dsma::api - Building router
Jun 15 15:58:31 linux-adm bbc5eb374230[115923] INFO dsma - Successfully connected to DSM at https://<fortanix_dsm_url>/
Jun 15 15:58:31 linux-adm bbc5eb374230[115923] INFO dsma - Listening on port: 8080#015
Jun 15 15:58:31 linux-adm bbc5eb374230[115923] INFO dsma - Certificate extn: pem#015
Jun 15 15:58:31 linux-adm bbc5eb374230[115923] INFO dsma - Attempting to construct TLS certificate.#015
Jun 15 15:58:31 linux-adm bbc5eb374230[115923] INFO dsma - Successfully constructed TLS certificate using the files found at certificate path.#015
Jun 15 15:59:01 linux-adm bbc5eb374230[115923] 2023-06-15 10:29:01.083Z eop1o8en 00000000 <= ::1:55276 POST /crypto/v1/sign#015
Jun 15 15:59:03 linux-adm bbc5eb374230[115923] INFO sdkms_provider::async_state - crypto/v1/keys/export, POST, X-Request-Id: QVWV1N-J, Status: 200 OK, Latency 359ms#015
Jun 15 15:59:03 linux-adm bbc5eb374230[115923] 2023-06-15 10:29:03.148Z eop1o8en 00000000 => 200 2065ms#015
Jun 15 16:00:03 linux-adm bbc5eb374230[115923] INFO valentino_lib::async_client - #015
Jun 15 16:00:03 linux-adm bbc5eb374230[115923] ==2023-06-15 10:30:03 An entry has been removed/updated. k: 2f5e259f-8ccc-4eb2-9ccf-d42728dd5d22, cause: Expired#015

Restart persistence is accomplished with this setup since the log forwarding continues even when the Docker container is restarted.

The following table lists the advantages and disadvantages of this procedure:

SL. NO	ADVANTAGES	DISADVANTAGES
1	This procedure uses a simple Docker configuration with fewer components in the log forwarding setup.	This procedure requires a remote Syslog server to be configured and running to accept incoming logs.
2	This procedure does not consume disk space since the container logs are not stored on the host filesystem.	This procedure requires a remote Syslog server to be configured to differentiate various incoming sources.
3	Container logs survive restarts and respawns using this setup.	Options for filtering log data are limited using this setup as compared to a log collector setup described in Export Logs to Syslog Server with Fluentd Daemon below. Since the container is directly shipping its logs over to a remote Syslog server, any filtering of useful logs has to be done on the receiving side.
4	Since logs are directly forwarded to a remote Syslog server, the latency would be lower on the other side using this setup.

4.0 Export Logs to Syslog Server with Fluentd Daemon

4.1 Deployment Type

This procedure involves running the Fortanix DSM Accelerator Webservice container application in a Docker system that is configured to forward logs to a Fluentd daemon running on the host and then forward logs to a Syslog server.

4.2 Setup

In this procedure, the Fortanix DSM Accelerator Webservice container logs will be directed to the fluentd daemon rather than being produced at the location: /var/lib/docker/containers/<container_id> so as to prevent disk space exhaustion.

Perform the following steps:

Configure the Fluentd driver in Docker using the configuration file: etc/docker/daemon.json:
```
{
  "log-driver": "fluentd",
  "log-opts": {
    "fluentd-address": "fluentdhost:port_nbr"
  }
}
```
Where, fluentdhost:port_nbr can be the IP address or the fully qualified domain name (FQDN) of the fluentd daemon running on that particular port number.
Alternatively, these configuration settings can also be provided when you run the Docker container using the following command:
```
sudo docker run -it --network host --log-driver fluentd --log-opt fluentd-address=host_ip:8888
```
NOTE
The configuration file settings will apply by default to all containers running with that Docker daemon on a host. A container can also be provided the arguments while running as described in the alternate way above to override the default settings.
Restart the Docker daemon using the following command so that the configuration file created in Step 1 comes into effect.
```
sudo systemctl restart docker
```
In this setup, a td-agent service, which is the fluentd daemon, is running on localhost:8888.
Install the fluentd output plugin, remote_syslog, on td-agent using the following command to successfully forward the container logs from fluentd to a remote Syslog server:
```
td-agent-gem install fluent-plugin-remote_syslog
```

Next, configure the fluentd agent to listen over TCP on port 8888 for the incoming Docker container logs and forward them to the rsyslog server. The configuration in the path etc/td-agent/td-agent.conf could be provided as follows:

# Logs from dsma-rws container
# TCP input
<source>
  @type forward
  @id input_tcp
  port 8888
  <parse>
    @type none
  </parse>
</source>
<match **>
  @type remote_syslog
  @id out_remote_syslog
  host rsyslog_server_ip
  port 514
  severity debug
  program fluentd
  hostname ${dsma}
  protocol tcp
  packet_size 65535
  <buffer dsma>
  </buffer>
  <format>
    @type out_file
</format>
</match>

When the Docker container is running with the above setup, the logs will be forwarded to the fluentd daemon, which in turn ships them over to the remote Syslog server. This can be observed by tailing the logs on the server side using the following command:

sudo tail -f /var/log/syslog

Sample output (trimmed):

Jun 16 04:39:31 linux-adm fluentd: 2023-06-16T04:39:26+05:{"container_id":"70c85b70d","container_name":"/focused_spence","source":"stdout","log":"INFO dsma::server - Initialized DsmAServer\r"}
Jun 16 04:39:31 linux-adm fluentd: 2023-06-16T04:39:27+05:{"source":"stdout","log":"INFO dsma::api - Building router\r","container_id":"70c85b70d","container_name":"/focused_spence"}Jun 16 04:39:31 linux-adm fluentd: 2023-06-16T04:39:29+05:{"log":"INFO dsma - Successfully connected to DSM at https://<fortanix_dsm_url>/\r","container_id":"70c85b70d","container_name":"/focused_spence","source":"stdout"}
Jun 16 04:39:31 linux-adm fluentd: 2023-06-16T04:39:29+05:{"container_id":"70c85b70d","container_name":"/focused_spence","source":"stdout","log":"INFO dsma - Listening on port: 8080\r"}
Jun 16 04:40:18 linux-adm fluentd: 2023-06-16T04:40:13+05:{"container_id":"70c85b70d","container_name":"/focused_spence","source":"stdout","log":"2023-06-15 23:10:13.791Z zmiK56Gh 00000000 <= ::1:59742 POST /crypto/v1/sign\r"}
Jun 16 04:40:18 linux-adm fluentd: 2023-06-16T04:40:15+05:{"container_id":"70c85b70d","container_name":"/focused_spence","source":"stdout","log":"INFO sdkms_provider::async_state - sys/v1/session/auth, POST, X-Request-Id: 94rqvTGX, Status: 200 OK, Latency 1641ms\r"}
Jun 16 04:40:18 linux-adm fluentd: 2023-06-16T04:40:15+05:{"container_id":"70c85b70d","container_name":"/focused_spence","source":"stdout","log":"INFO sdkms_provider::async_state - crypto/v1/keys/export, POST, X-Request-Id: l4to4daL, Status: 200 OK, Latency 257ms\r"}
Jun 16 04:40:18 linux-adm fluentd: 2023-06-16T04:40:15+05:{"container_id":"70c85b70d","container_name":"/focused_spence","source":"stdout","log":"2023-06-15 23:10:15.698Z zmiK56Gh 00000000 => 200 1906ms\r"}

Restart persistence is accomplished with this setup since the log forwarding continues even when the Docker container is restarted.

The following table lists the advantages and disadvantages of this procedure:

SL. NO.	ADVANTAGES	DISADVABTAGES
1	Using this procedure, a Docker container can be configured with different logging drivers as per the use case.	This procedure involves many components in the log forwarding setup. Hence, configurations would be more complex.
2	Since the container logs are not stored in the host’s filesystem, this procedure does not consume disk space.	This procedure requires a remote Syslog server to be configured and running to accept incoming logs.
3	With this procedure, the container logs survive restarts and respawns.	With this setup, in case of network failure or connection error to the remote Syslog server, the fluentd log collector’s buffer size should be sufficient to ensure enough queuing.
4	Using this procedure, the fluentd log-collector can filter or restructure incoming logs before forwarding them to a remote Syslog server.

NOTE
Even with remote logging drivers configured, users can troubleshoot and review Fortanix DSM Accelerator Webservice container logs on the host without having to always reach out to the SIEM team. The Docker supports dual logging, which allows you to use the Docker logs command to read the container logs regardless of the configured logging drivers. This is enabled by default but is configurable and comes with some default constraints such as maximum size, and so on.

For example, run the following command to tail the container logs for a remote logging driver container, 4d1cc8da71fb:

sudo docker logs -f 4d1cc8da71fb

4.3 References

Docker official documentation

4.4 Failover Condition - Network Issues

Problem Statement:

In case the remote Syslog server is unreachable due to network issues (or any other reason), will the setup have some buffer where the incoming container logs will be queued, and once the connection is established, will you still receive the pending logs?

Observations:

Fortanix simulated this failover by temporarily disabling rsyslog [socket+service] with fluentd trying to establish a connection and observed that fluentd adds the container logs into a buffer, which is periodically flushed to the remote server. In case the remote connection is not established, fluentd retries every few minutes while maintaining and adding to the same log buffer. After restarting the rsyslog service, the fluentd retry succeeded, and it flushed the log buffer chunk over to the rsyslog server. On tailing remote server logs, container logs were seen from start to the last API call.