A SaltStack Highstate Killswitch

On rare occasion it’s necessary to debug a condition on a system by making temporary changes to the running system. If you’re using config management, especially as part of your deployment process, it’s necessary to disable it so that your temporary changes won’t be reset. salt-call doesn’t natively have a mechanism for this like Puppet does (puppet agent –disable; puppet agent –enable). It’s possible to do this yourself, though.

This requires that you’re using the failhard option in your configuration, that you’re using the 2014.7 (Helium) or above release, and also assumes you have some base state that is always included and is always included first.

First add two states:

highstate/enable.sls:

Ensure highstate is enabled:
  grains.present:
    - name: highstate_disabled
    - value: False

highstate/disable.sls:

Ensure highstate is disabled:
  grains.present:
    - name: highstate_disabled
    - value: True

In your base state:

{% if salt['grains.get']('highstate_disabled', False) %}
Exit if highstate is disabled:
  test.fail_without_changes:
    - name: "Salt highstate is disabled. To re-enable, call 'salt-call state.sls highstate.enable'."
{% endif %}

Now, let’s test it!

# salt-call state.sls highstate.disable
local:
  Name: highstate_disabled - Function: grains.present - Result: Changed

Summary
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1

# salt-call state.highstate
local:
----------
          ID: Exit if highstate is disabled
    Function: test.fail_without_changes
        Name: Salt highstate is disabled. To re-enable, call 'salt-call state.sls highstate.enable'.
      Result: False
     Comment: Failure!
     Started: 00:42:20.384504
     Duration: 1 ms
     Changes:

Summary
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1

# salt-call state.sls highstate.enable
local:
  Name: highstate_disabled - Function: grains.present - Result: Changed

Summary
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1

# salt-call state.highstate
local:

Summary
------------
Succeeded: 0
Failed:    0
------------
Total states run:     0

Note in the above that I’m using a couple settings that make my output strange:

# Show terse output for successful states and full output for failures.
state_output: mixed

# Only show changes
state_verbose: False

So, a run that doesn’t change anything will show that it didn’t run anything. Total states run: 0 is a successful highstate run in that situation.

Something to note about this solution is that it will only disable highstate and may not disable state.sls, state.template, or other state calls, if the initial state isn’t included. This obviously won’t disable any other module call either.