Splunk Integration with Sensu Server

Introduction

This guide describes the steps to integrate Splunk with an existing Sensu server.

Integration Steps

1. List the existing handlers.

   sensuctl handler list

   

2. Create a TCP handler to send the events to external systems such as Splunk.

   sensuctl handler create event-storage --type set --handlers splunk_tcp

   

3. Now edit the event-storage to save the TCP handler configuration.

   sensuctl edit handler event-storage

   This will open the file in a file editor.

4. Update the host to localhost and port to 7758. Press the “ESC” command and then “SHIFT+:WQ!” to save the file.  

   
   

5. To list the newly created handler, run the following command.

   sensuctl handler list

   

6. To list all the checks on the Sensu servers, run the following command.

   sensuctl check list

   

   You will see all the checks (cpu-usage, disk-usage and so on.) are added to the email handlers. Now you need to add splunk_tcp handler created in Step 2 and 3 so that the events can be sent to the TCP handler.

7. Now edit the check to add splunk_tcp handler.

   sensuctl edit check cpu-usage

   Add “splunk_tcp” under handlers and save the file.  

   

   Similarly, edit all the checks and add the splunk_tcp handler.

8. You can check locally using Ncat if the events are being sent to port 7758 (if the nc command is not available, the package can be installed with “#yum install nc”.

   nc -v -l -p 7758

   

   You can see the JSON output of the event.

9. Download and install the Splunk forwarder on the Sensu server to forward the events to Splunk.

   wget -O splunkforwarder-8.2.0-e053ef3c985f-linux-2.6-x86_64.rpm 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.2.0&product=universalforwarder&filename=splunkforwarder-8.2.0-e053ef3c985f-linux-2.6-x86_64.rpm&wget=true'

   

10. Create a file called “inputs.conf” under “/opt/splunkforwarder/etc/system/local” directory and add the configuration below.

    [tcp://:7758]
    index=sensu   [Make sure the index name “sensu” present in Splunk]

    

11. Start Splunk forwarder.

    /opt/splunkforwarder/bin/splunk start --accept-license

    The command above will prompt for username and password, enter the username and password.

12. Configure forwarder connection to the index server.

    /opt/splunkforwarder/bin/splunk add forward-server hostname.domain:7758

    Replace the hostname.domain name with Splunk index server name or IP, in this example we are using Splunk server with IP 52.188.83.113.  

    

    NOTE

    Make sure port 7758 is opened and added to receive data.

    1. Log in to Splunk and go to Settings.

    2. Under the Data section, select Forwarding and receiving.

    3. Click Receiving data (add new) and enter port 7758 and click Save.

       

13. Test the forwarder connection.

    /opt/splunkforwarder/bin/splunk list forward-server

    

14. Log in to the Splunk console and check if the events are being sent to the index.