Splunk Integration with Sensu Server

1.0 Introduction

This article describes the procedure to integrate Splunk with an existing Sensu server.

2.0 Configuring Splunk Integration

Perform the following steps in the same order as mentioned to configure Splunk integration with the Sensu server.

2.1 List Existing Handlers

Run the following command to list the existing handlers:

sensuctl handler list

2.2 Create a TCP Handler

Run the following command to create a TCP handler to send events to external systems, such as Splunk:

sensuctl handler create event-storage --type set --handlers splunk_tcp

2.3 Edit the TCP Handler Configuration

1. Run the following command to edit the newly created event-storage handler:

   sensuctl edit handler event-storage

   

   
   This will open the configuration file in the default editor.

2. Update the host to localhost and port to 7758.

3. Press ESC and then SHIFT+:WQ! to save the file.  

   

2.4 Verify Handlers

Run the following command to list the newly created handler:

sensuctl handler list

2.5 List Checks on the Sensu Server

Run the following command to list all the checks on the Sensu server:

sensuctl check list

You will see all the checks associated with email handlers. For example, cpu-usage, disk-usage and so on.

Add the newly created splunk_tcp handler so that events can also be sent to Splunk.

2.6 Add the splunk_tcp Handler to Checks

1. Run the following command to edit the check to add splunk_tcp handler:

   sensuctl edit check cpu-usage

2. Add splunk_tcp under handlers section.

   

3. Save and exit.

4. Repeat this step for all required checks.

2.7 Validate Events Locally (Optional)

Run the following command to verify if events are being sent to port 7758 :

nc -v -l -p 7758

If nc is not available, install it using:

yum install nc

You should see the JSON event output.

2.8 Install Splunk Forwarder

Run the following command to download and install the Splunk Universal Forwarder on the Sensu server:

wget -O splunkforwarder-8.2.0-e053ef3c985f-linux-2.6-x86_64.rpm 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.2.0&product=universalforwarder&filename=splunkforwarder-8.2.0-e053ef3c985f-linux-2.6-x86_64.rpm&wget=true'

2.9 Configure Splunk Forwarder Input

1. Create a file named inputs.conf under /opt/splunkforwarder/etc/system/local/.

2. Add the following configuration:

   [tcp://:7758]
   index=sensu   [Make sure the index name “sensu” present in Splunk]

   

2.10 Start Splunk Forwarder

Run the following command to start the Splunk forwarder service:

/opt/splunkforwarder/bin/splunk start --accept-license

When prompted, enter the Splunk username and password.

2.11 Configure Forwarder Connection

Run the following command to add the Splunk index server as a forwarder:

/opt/splunkforwarder/bin/splunk add forward-server hostname.domain:7758

Replace <hostname.domain> with the Splunk index server name or IP address.

For example:

/opt/splunkforwarder/bin/splunk add forward-server 52.188.83.113:7758

2.12 Test Forwarder Connection

Run the following command to verify the forwarder connection:

/opt/splunkforwarder/bin/splunk list forward-server

2.13 Verify Events in Splunk

1. Log in to the Splunk console.

2. Navigate to the configured index.

3. Verify that Sensu events are being received.