Raspberry Pi as a Splunk Universal Forwarder

Raspberry Pi as a Splunk Universal Forwarder

Here is the overall process:

1. Build the core splunk system (the indexer & search head) and confirm it works ok. I’m not going to cover that here.
2. Set your core splunk system to receive traffic from forwarders
3. Install Raspbian on the Raspberry Pi (from raspberrypi.org)
4. Using raspi-config, set the disk to use the full SD card, set the hostname, and set the timezone
5. Set your router to give a fixed ip to the raspberry pi.
6. Set up a USB stick to be the /var/log directory
   1. format the USB stick to ext3
   2. move the /var/log dir to the USB stick (instructions here are for moving /var, but the procedure is the same).
      Note, in the step to edit the fstab file, I used the following:
      UUID=uuid  /var/log  ext3   defaults   0   1
      (use blkid to determine the UUID)
7. Install syslog-ng (sudo apt-get install syslog-ng)
8. Configure a system to send syslogs to this system using udp port 514.
9. Configure syslog-ng
   1. edit syslog-ng.conf (sudo vi /etc/syslog-ng/syslog-ng.conf)
   2. add the following lines to the appropriate sections to set syslog-ng to listen for syslogs on udp port 514 and save them to /var/log/udp514.log (or whatever you want to call your log file. syslog-ng can do a lot more if you wish, including create unique log files for every log that comes in)

       # source for syslog 514 traffic
       source s_udp514 { udp(port(514)); };
       # destination for udp 514 syslogs
       destination d_udp514 { file("/var/log/udp514.log"); };
       # All udp514 logs
       log { source(s_udp514); destination(d_udp514); };

   3. restart syslog-ng (sudo /etc/init.d/syslog-ng restart)
   4. watch that log to see it is getting data (tail -f /var/log/udp514.log)
10. Install the Splunk Universal Forwarder
    1. download the forwarder
    2. install the forwarder (sudo tar xvzf forwarder-for-linux-arm-raspberry-pi_10.tgz -C /opt)
    3. configure splunk to run with user id splunk & start splunk
       sudo useradd splunk
       sudo groupadd splunk (the group may be created already)
       sudo chown splunk:splunk /opt/splunkforwarder/
       sudo -H -u splunk /opt/splunkforwarder/bin/splunk start
    4. configure to run at boot
        sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk
       
    5. make sure any logs you wish to forward are readable by splunk. Since logs are typically read-writefor the owner and read-only for the group, you can change the group.  You may choose to do a single file, eg
       sudo chgrp splunk /var/log/udp514.log
       or all the log files, eg
       sudo -R chgrp splunk /var/log/
11. Reboot your pi and confirm splunk is running using the right id (splunk)
    sudo reboot -r now
    ps -ef | grep splunk
    You should  get a result similar to this, showing that splunk, not root, is running splunk (the first column is the user):

    splunk    2188     1 24 00:21 ?        00:00:08 splunkd -p 8089 start
    splunk    2189  2188  0 00:21 ?        00:00:00 [splunkd pid=2188] splunkd -p 8089 start [process-runner]
    pi        2262  2247  0 00:22 pts/0    00:00:00 grep --color=auto splunk

12. Configure the Universal Forwarder
    1. since splunk now runs as the splunk id, change to that id and change to the splunk directory
       sudo su – splunk
       cd /opt/splunkforwarder/bin
    2. set the admin password to something unique (the default is “changeme”)
       ./splunk edit user admin -password -role admin -auth admin:changeme
    3. set the forwarder to forward (use your new password)
       ./splunk add forward-server : -auth admin:
    4. set what to monitor and forward
       ./splunk add monitor /var/log/
13. Log into your Splunk instance and check out your logs!
    1. If you have problems, check out this troubleshooting page: Troubleshooting Forwarding
    2. Validate the approach works by shutting down your core Splunk.  Notice that the next time you power it up, after a little delay logs will start filling in from during your outage.
14. Configure your logs to rotate
    1. Edit /etc/logrotate.conf and add the following (this will rotate when the size hits 1G, the new log file it creates will be owned by splunk/splunk with 740 permissions, and we’ll keep up to 10 files)

       /var/log/udp514.log {
           size 1g 
           create 740 splunk splunk
           rotate 10
       }

    2. Note that Splunk won’t read the rotated logs, so it probably makes sense to zip them and keep fewer copies.

 Fuente: https://stichintime.wordpress.com/2015/03/23/raspberry-pi-as-a-splunk-universal-forwarder-to-store-and-foward-logs/