Splunk saved search state and execution module support in SaltStack

We (Lyft) believe strongly in the concept of infrastructure as code. If it isn’t automated, it isn’t finished. This belief also applies to our monitoring and alerting. We’re using Splunk saved searches for portions of our alerting and want to ensure that our developers can quickly and easily define alarms in a standard way to be able to share alarms between services.

We’ve added the splunk_search execution module and splunk_search state module to the 2015.2 Saltstack release (in release candidate status at the time of this writing) so that we can manage our searches via orchestration.

This lets us define a saved search with an alarm, as a state, like so:

Manage splunk search {{ grains.service_group }} no call volume:
  - name: {{ grains.service_group }} no call volume
  - action.email.format: plain
  - action.email.inline: '1'
  - action.email.sendresults: '1'
  - action.email.to: {{ grains.service_group }}@myorg.pagerduty.com
  - actions: email
  - alert.expires: 1d
  - alert.severity: '4'
  - alert.suppress: '1'
  - alert.suppress.period: 30m
  - alert.track: '1'
  - alert_comparator: greater than
  - alert_threshold: '0'
  - alert_type: number of events
  - cron_schedule: '*/5 * * * *'
  - description: '**MANAGED BY ORCHESTRATION** Fires when {{ grains.service_group }} has no volume for X minutes'
  - dispatch.earliest_time: -6m
  - dispatch.latest_time: -1m
  - dispatch.ttl: 1p
  - is_scheduled: '1'
  - search: 'index=* source="*access.log" host="{{ grains.service_group }}*" | regex method="GET|POST|PUT|DELETE" | stats count as count | where count = 0'

This saved search will send an alert to pager duty if the service has no call volume for a period of time across all of its nodes.

Like all of our modules, these were written mostly for our use-cases, but we hope they’re useful to you as well. Please contribute back if there are any features you need!

Want to help us write and upstream software like this? Apply for a position at Lyft. If you want to work directly with me, apply for a DevOps Engineer, Senior DevOps Engineer, or Senior Platform Engineer position.