Concurrent and Queued SaltStack State Runs

The function "state.highstate" is running as PID 17587 and was started at 2014, Aug 29 23:21:46.540749 with jid 20140829232146540749

Ever get an error like that? Salt doesn’t allow more than a single state run to occur at a time, to ensure that multiple state runs can’t interfere with each other. This is really important, for instance, if you run highstate on a schedule, since a second highstate might be called before the previous one had finished.

What if you use states as simple function calls, for frequent actions, though? Or what if you’re using Salt for orchestration and want to run multiple salt-calls for a number of different salt state files concurrently?

In the Salt 2014.7 (Helium) release it’s possible to specify that you want a state run to occur concurrently:

# salt-call state.sls mystate concurrent=True

Let’s assume you do need to ensure two Salt runs don’t occur concurrently, but rather than having the command fail you want to wait until the first run finishes. As of the 2014.7 release this is also possible via the queue argument:

# salt-call state.sls mystate queue=True

As I was writing this post a thread was started on the salt users group where this subject was brought up, and it seems at the time of this writing that concurrent isn’t working properly for Windows minions. Hopefully that will get fixed before the 2014.7 release, but if not, just keep an eye out for Windows support.


Update (September 2, 2014):

As Colton mentions in the comments, using concurrent is dangerous, since Salt’s state system isn’t designed to be run concurrently. This doesn’t mean that you can’t use concurrent, if you use it in a way you know is safe.

I use concurrent for things I know won’t conflict, such as a state file that calls a set of execution modules to make a simple access point to a complex set of standard actions, like registering or deregistering an ELB. Concurrent can be really important for this, if a Salt run hangs on a node, and you need to deregister the node.

Another example of where I use concurrent is for the boto_* state modules, for orchestration. I know two services won’t have state definitions that conflict, since they reference completely different infrastructure.

So, that said, the code is marked with a large warning statement so you may be told to not use concurrent if you’re using it in a way that’s dangerous, when you open issues in github.

  • Note that queue is much safer than concurrent, and is the recommended method here. There’s a giant warning under concurrent:

    https://github.com/saltstack/salt/blob/develop/salt/modules/state.py#L356-L359

  • Hi !

    Thanks a lot for your post ! I was looking for these options and never found them ! We are several people working on the same Salt master and it’s the war when one want to launch a process and another one too. :p

    Regards,

  • Andrew Hammond

    I’m trying to figure out how to integrate queue=True into my reactor driven orchestrations. I’ve tried the following and none of them work. Any suggestions?

    foo_arg:
    salt.state:
    – tgt: some_host
    – highstate: True
    – arg:
    – queue=True

    foo_kwarg:
    salt.state:
    – tgt: some_host
    – highstate: True
    – kwarg:
    queue: True

    foo_kwargs:
    salt.state:
    – tgt: some_host
    – highstate: True
    – kwargs:
    queue: True