For the Salt 2014.7 released we (Lyft) upstreamed a number of Salt execution and state modules for AWS. These modules manage various AWS resources. For most of the resources you’ll want to create, you’ll probably want to add cloudwatch alarms to go along with them. It’s not really difficult to do:
Ensure myelb exists:
boto_elb.present:
- name: myelb
- subnets:
- subnet-12345
- subnet-12346
- subnet-12347
- security_groups:
- elb
- listeners:
- elb_port: 443
instance_port: 80
elb_protocol: HTTPS
instance_protocol: HTTP
certificate: 'arn:aws:iam::1234:server-certificate/mycert'
- health_check:
target: 'HTTP:80/'
- attributes:
cross_zone_load_balancing:
enabled: true
connection_draining:
enabled: true
timeout: 30
- profile: myprofile
Ensure myelb ELB UnHealthyHostCount alarm exists:
boto_cloudwatch_alarm.present:
- name: 'myelb ELB UnHealthyHostCount **MANAGED BY SALT**'
- attributes:
metric: UnHealthyHostCount
namespace: AWS/ELB
statistic: Average
comparison: '>='
threshold: 1.0
period: 600
evaluation_periods: 6
unit: null
description: myelb ELB UnHealthyHostCount >= 1
dimensions:
LoadBalancerName: [myelb]
alarm_actions: ['arn:aws:sns:us-east-1:12345:alarm-myservice']
ok_actions: ['arn:aws:sns:us-east-1:12345:alarm-myservice']
insufficient_data_actions: []
- profile: myprofile
It’s not difficult, but there’s a few problems here:
- It’s really verbose.
- You have to remember to do it for every resource.
- For most (or all) of your ELBs, you’ll probably want the exact same alarm.
We believe that it should be easy to do the right thing and difficult to do the wrong thing. In SaltStack 2015.2 (in feature freeze at the time of this writing) we’ve made it possible to have some resources manage their own alarms. Even better, we’ve also made it possible to define these alarms through pillars, and for each resource type we use a default pillar key, so that you can define an alarm once and have it apply automatically for all resources of that type.
For example, here’s how to define an ELB resource to manage its alarm:
Ensure myelb exists:
boto_elb.present:
- name: myelb
- subnets:
- subnet-12345
- subnet-12346
- subnet-12347
- security_groups:
- elb
- listeners:
- elb_port: 443
instance_port: 80
elb_protocol: HTTPS
instance_protocol: HTTP
certificate: 'arn:aws:iam::1234:server-certificate/mycert'
- health_check:
target: 'HTTP:80/'
- attributes:
cross_zone_load_balancing:
enabled: true
connection_draining:
enabled: true
timeout: 30
- alarms:
UnHealthyHostCount:
name: 'ELB UnHealthyHostCount **MANAGED BY SALT**'
attributes:
metric: UnHealthyHostCount
namespace: AWS/ELB
statistic: Average
comparison: '>='
threshold: 1.0
period: 600
evaluation_periods: 6
unit: null
description: ELB UnHealthyHostCount >= 1
alarm_actions: ['arn:aws:sns:us-east-1:12345:alarm-myservice']
ok_actions: ['arn:aws:sns:us-east-1:12345:alarm-myservice']
insufficient_data_actions: []
- profile: myprofile
The name defined for the alarm will have the ELB name prepended, to ensure uniqueness.
If we want the same alarm to apply to multiple ELBs, we can move it to a pillar:
my_elb_alarms:
UnHealthyHostCount:
name: 'ELB UnHealthyHostCount **MANAGED BY SALT**'
attributes:
metric: UnHealthyHostCount
namespace: AWS/ELB
statistic: Average
comparison: '>='
threshold: 1.0
period: 600
evaluation_periods: 6
unit: null
description: ELB UnHealthyHostCount >= 1
alarm_actions: ['arn:aws:sns:us-east-1:12345:alarm-myservice']
ok_actions: ['arn:aws:sns:us-east-1:12345:alarm-myservice']
insufficient_data_actions: []
Then we can define the ELB resource like so:
Ensure myelb exists:
boto_elb.present:
- name: myelb
- subnets:
- subnet-12345
- subnet-12346
- subnet-12347
- security_groups:
- elb
- listeners:
- elb_port: 443
instance_port: 80
elb_protocol: HTTPS
instance_protocol: HTTP
certificate: 'arn:aws:iam::1234:server-certificate/mycert'
- health_check:
target: 'HTTP:80/'
- attributes:
cross_zone_load_balancing:
enabled: true
connection_draining:
enabled: true
timeout: 30
- alarms_from_pillars: my_elb_alarms
- profile: myprofile
boto_elb_alarms is a special pillar key. If it’s defined, all boto_elb resources will use it. So, if we change the pillar key to that, we can remove the ‘alarms_from_pillars’ line from the ELB resource and the alarm will still apply.
Notice in the above alarm that alarm_actions and ok_actions specify ARNs for a specific service? If you’re defining the alarms for multiple services, it’s necessary to make that specific to the service.
Our orchestration is a bit custom and we call a set of states for each service, passing in environment variables, which set custom grains. We use those grains in the pillar defined alarms, so that we can generically set the service name in the pillars. However, you don’t need to have our custom implementation to make this work for you. It’s possible to override values in the alarms:
Ensure myelb exists:
boto_elb.present:
- name: myelb
- subnets:
- subnet-12345
- subnet-12346
- subnet-12347
- security_groups:
- elb
- listeners:
- elb_port: 443
instance_port: 80
elb_protocol: HTTPS
instance_protocol: HTTP
certificate: 'arn:aws:iam::1234:server-certificate/mycert'
- health_check:
target: 'HTTP:80/'
- attributes:
cross_zone_load_balancing:
enabled: true
connection_draining:
enabled: true
timeout: 30
- alarms:
UnHealthyHostCount:
attributes:
alarm_actions: ['arn:aws:sns:us-east-1:12345:alarm-mycustomservice']
ok_actions: ['arn:aws:sns:us-east-1:12345:alarm-mycustomservice']
In the above, only alarm_actions and ok_actions will be overridden, letting you set this on any ELB that needs to have a non-default value.
For the 2015.2 release we’ve only added support for boto_elb and boto_asg, but we plan on adding support for a number of the other boto_* modules in future releases.