Truly ordered execution using SaltStack (Part 2)

A while back I wrote a post about sequentially ordered SaltStack execution. 2014.7 (Helium) has been released and the listen/listen_in feature I described is now generally available. It’s been about 6 months since I’ve been using Salt in a sequentially ordered manner and there’s some other patterns I’ve picked up here. Particularly, there’s a couple gotchas to watch out for: includes and Jinja.

Includes imply a requirement between modules. Requirements can modify ordering, so it’s important to be strict about how you handle them. For example, when reading the following, remember that include implies require:

top.sls:

base:
  '*':
    - base
    - service

service.sls:

include:
  - apache
  - redis

apache.sls:

include:
  - vhost

So, the order you’re getting here is: base, vhost, apache, redis, service. If you remember this is the behavior, then it isn’t actually difficult when you’re reading the file. Because of this, I recommend always placing the include statement at the top of a state, if you’re going to use it mixed in with other states. This is important, because this won’t do what you assume:

Ensure apache is installed:
  pkg.installed:
    - name: apache2

include:
  - vhost

Ensure apache is started:
  service.running:
    - name: apache2

In the above, if the vhost module required the apache2 package to be installed before it can run, it’ll fail. This is because even if the include is placed after the package state, it’s still being included before it, because the vhost module is a requirement for the current module.

What can we do if we need this behavior, then? Jinja to the rescue:

Ensure apache is installed:
  pkg.installed:
    - name: apache2

{% include 'vhost.sls' %}

Ensure apache is started:
  service.running:
    - name: apache2

When doing includes via Jinja, the file is simply being included from this module’s context. The contents of the vhost.sls file will be placed into that location. This occurs before further Jinja evaluation as well, so this is a completely safe way to handle this situation. Of course, it would be nice for salt to have a native way of handling that, so I have an open issue for this.

Note that the syntax for the Jinja include is different. Rather than using . for separation, Jinja includes use / and the .sls extension is required. Like state includes, the Jinja includes start from the file root, so apache.vhost would be “apache/vhost.sls”.

Jinja itself is something to pay attention to as well. Jinja is always evaluated before the state execution. This is useful in a number of ways (for instance, conditionally including or excluding large portions of code, or looping a bunch of states together), but it’s also a bit confusing when you are considering ordering. For instance, this won’t work:

Ensure myelb exists:
  boto_elb.present:
    - name: myelb
    - availability_zones:
        - us-east-1a
    - listeners:
        - elb_port: 80
          instance_port: 80
          elb_protocol: HTTP
        - elb_port: 443
          instance_port: 80
          elb_protocol: HTTPS
          instance_protocol: HTTP
          certificate: 'arn:aws:iam::123456:server-certificate/mycert'
    - health_check:
        target: 'TCP:80'
    - profile: myprofile

{% set elb = salt['boto_elb.get_elb_config']('myelb', profile='myprofile') %}

Ensure myrecord.example.com cname points at ELB:
  boto_route53.present:
    - name: myrecord.example.com.
    - zone: example.com.
    - type: CNAME
    - value: {{ elb.dns_name }}
    - profile: myprofile

When you read this in order, it looks completely logical. However, the Jinja is always going to be executed before the states, so the elb variable is going to be set to None, then the ELB will be created, then the route53 record will fail to be created.

This is a contrived example, since the boto_elb.present function will handle route53 on your behalf, but it illustrates an issue you’ll need to watch out for. Always remember Jinja will execute first and protect against it.