Dealing with splunkforwarder via Config Management

The splunkforwarder package is very poorly written, at least for Debian/Ubuntu. There’s a number of things it does that make it difficult to use:

  1. It installs a splunk user and group, but doesn’t install them as system users/groups, so they’ll conflict with your uids/gids.
  2. It requires manual interaction the first time you start the daemon, on every single system it’s installed on.
  3. It modifies its configuration files when the daemon restarts.

The first is an honest mistake, but the last two put me into a blind rage. There’s not great documentation about how to workaround this, so to avoid other opsen going into rages here’s how you can handle this shitty package:

  1. Add a splunk user/group as a system user before the package is installed.
  2. Install the package.
  3. Replace the init script (I have one you can use below).
  4. Use a consistent hash for the SSL password. This won’t completely avoid the rewriting of the config files, but it’ll avoid the more common case where a rewrite will occur.
  5. Start the daemon.

Replacing the init script

Any single Splunk command you initially run will require that you accept a license. If you’re using config management, it’s likely that it’s going to use ‘service status’ before it does ‘service start’ or ‘service restart’. This means it’s necessary to add ‘–no-prompt –answer-yes –accept-license’ to each relevant command. Just to be safe lets just always call it.

# /etc/init.d/splunk
# init script for Splunk.
# generated by 'splunk enable boot-start'.
# Provides: splunkd
# Required-Start: $remote_fs
# Required-Stop: $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start splunk
# Description: Splunk indexer service

splunk_start() {
  echo Starting Splunk...
  "/opt/splunkforwarder/bin/splunk" start --no-prompt --answer-yes --accept-license
splunk_stop() {
  echo Stopping Splunk...
  "/opt/splunkforwarder/bin/splunk" stop --no-prompt --answer-yes --accept-license
splunk_restart() {
  echo Restarting Splunk...
  "/opt/splunkforwarder/bin/splunk" restart --no-prompt --answer-yes --accept-license
splunk_status() {
  echo Splunk status:
  "/opt/splunkforwarder/bin/splunk" status --no-prompt --answer-yes --accept-license
case "$1" in
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1

exit $RETVAL

Use a consistent hash for the SSL password

This is insanely annoying. The SSL password is just used for Splunk to access its SSL key. The splunk daemon will convert any clear-text password in its configuration files to a hashed password, based off a secret that is generated per-host. Obviously Splunk is ticking off a Government compliance checklist with this stupidity. There is absolutely no security benefit to what Splunk is doing here. Not only does Splunk change this line, but it also reorders the file, assuring there’s no sane way to handle this via config management.

Thankfully, it’s possible to pre-set the secret that’s being used in /opt/splunkforwarder/etc/auth/splunk.secret. On a test node, generate some long secret, place it into that file, then set your cleartext ssl password (which is password, by default) and restart Splunk. It’ll generate a hash and rewrite your configuration file. Now, take that hash and your generated secret and use them in your configuration management. Since you have a stable secret across your nodes and a hash that was generated from it, Splunk won’t change it.