by Ryan Lane

Custom service to service authentication using IAM/KMS

Similar to abusing IAM and STS, we can also abuse IAM and KMS to let Amazon do our service-to-service authentication for us. Unlike STS, though, KMS is almost perfect for this use case.

Let’s recap a bit from the STS post, though. What I’m aiming for is service to service authentication with the following specs:

  1. Has no chicken and egg trust problem. Re-use AWS to provide the chicken, we’ll use it to lay the eggs.
  2. Can be used only from one service to another service. The service receiving the token shouldn’t be able to reuse the token to impersonate the sender.
This is a preview of Custom service to service authentication using IAM/KMS. Read the full post (1821 words, estimated 7:17 mins reading time)
by Ryan Lane

Custom service to service authentication using IAM/STS

Lately I’ve wanted to be able to use IAM directly for authentication. Specifically, I wanted a way for a service to be able to verify that a request from another service was from a particular IAM role and that the request’s auth is still valid. I want this because I want to avoid the chicken and egg problem of bootstrapping auth systems in the cloud. You need some secret on your instances that allow it to talk to an auth system to do authentication between services. Amazon already provides this in the form of IAM roles. Unfortunately, only AWS services have access to verify these credentials. However, it’s possible to abuse IAM and STS to achieve this goal.

This is a preview of Custom service to service authentication using IAM/STS. Read the full post (1345 words, estimated 5:23 mins reading time)
by Ryan Lane

Using development branch SaltStack python modules in the stable release

SaltStack has a lot of development work occurring and sometimes you want early access to features. Thankfully, for a number of Salt’s features it’s possible to very easily bring them into the stable release without needing to maintain a fork or a non-standard version.

Salt is built to be very modular, so that you can write/run your own custom modules. This same mechanism works for overriding the core modules as well. I’ll explain how this works in a masterless setup (since that’s what I use), but the process is pretty similar for master/minion setups as well.

This is a preview of Using development branch SaltStack python modules in the stable release. Read the full post (375 words, estimated 1:30 mins reading time)
by Ryan Lane

KMS support added to SaltStack Beryllium (development) branch

In the development branch of SaltStack (to be the Beryllium release) I’ve added the boto_kms state and execution modules. This allows you to manage KMS master keys, their policies, key rotation, and other attributes via states. It also allows you to make KMS calls from other state and execution modules.

Here’s an example of managing a key and its attributes via a state:

This is a preview of KMS support added to SaltStack Beryllium (development) branch. Read the full post (354 words, estimated 1:25 mins reading time)
by Ryan Lane

Investigating local queuing: Redis, NSQ and LMDB

Systems designed for cloud services assume instances can die at any time, so they’re written to defend against this. It’s important to also remember that networks in cloud services are also incredibly unreliable, and often much less reliable than the instances themselves. When considering a design, it’s important to remember that a node can be partitioned from other services and possibly for long periods of time.

One easy consideration here is logs (including stats and analytics events). We want to ensure delivery of logs, but we also don’t want delivery to affect service operation.

This is a preview of Investigating local queuing: Redis, NSQ and LMDB. Read the full post (1018 words, estimated 4:04 mins reading time)
by Ryan Lane

DynamoDB support in SaltStack 2015.2

In the 2015.2 SaltStack release we’ve added the boto_dynamodb execution module and boto_dynamodb state module. This allows you to create DynamoDB tables via states:

Ensure DynamoDB table exists:
  boto_dynamodb.present:
    - table_name: {{ grains.cluster_name }}
    - read_capacity_units: 10
    - write_capacity_units: 10
    - hash_key: id
    - hash_key_data_type: S
    - global_indexes:
      - data_type_date_index:
        - name: "data_type_date_index"
        - read_capacity_units: 10
        - write_capacity_units: 10
        - hash_key: data_type
        - hash_key_data_type: S
        - range_key: modified_date
        - range_key_data_type: S
      - data_type_revision_index:
        - name: "data_type_revision_index"
        - read_capacity_units: 10
        - write_capacity_units: 10
        - hash_key: data_type
        - hash_key_data_type: S
        - range_key: revision
        - range_key_data_type: N

Note that at this point the module will only create and delete tables and indexes. It doesn’t currently support dynamically adding and removing indexes or changing read or write capacity units. These are features we’d love to see, and will likely add in the future. If you’d like to beat us to this, please send in pull requests!

This is a preview of DynamoDB support in SaltStack 2015.2. Read the full post (238 words, estimated 57 secs reading time)
by Ryan Lane

Splunk saved search state and execution module support in SaltStack

We (Lyft) believe strongly in the concept of infrastructure as code. If it isn’t automated, it isn’t finished. This belief also applies to our monitoring and alerting. We’re using Splunk saved searches for portions of our alerting and want to ensure that our developers can quickly and easily define alarms in a standard way to be able to share alarms between services.

We’ve added the splunk_search execution module and splunk_search state module to the 2015.2 Saltstack release (in release candidate status at the time of this writing) so that we can manage our searches via orchestration.

This is a preview of Splunk saved search state and execution module support in SaltStack. Read the full post (372 words, estimated 1:29 mins reading time)
by Ryan Lane

Grafana dashboard orchestration using SaltStack

As mentioned in my post on Cloudwatch alarms, we (Lyft) believe that it should be easy to do the right thing and difficult to do the wrong thing. We operate on the concept “If you build it, you run it.” Running your own service isn’t easy, if you don’t have the right tools to help you, though.

We’re using Graphite and Grafana for time series data dashboards. With a consistent configuration management pattern all new services start with their data flowing into Graphite. Dashboard management is tricky, though. We encourage teams to add custom metrics to their services and use them in panels and rows for their services, but we also want to provide a number of consistent panels/rows for all services. We also want to avoid making teams go between multiple dashboards to monitor their own services.

This is a preview of Grafana dashboard orchestration using SaltStack. Read the full post (838 words, estimated 3:21 mins reading time)
by Ryan Lane

SaltConf15: Sequentially Ordered Execution in SaltStack talk and slides

Here’s another talk that I gave at SaltConf15. It’s about sequentially ordered Salt and if you’ve read my blog posts on it, this probably won’t add a lot of technical info, but it’ll give a lot more context behind why you’d want to use Salt in a sequentially ordered way. Enjoy!

Sequentially Ordered Execution in SaltStack

Here’s the slides.. Note: though my blog is creative commons licensed, the slides are all rights reserved (sorry!).

This is a preview of SaltConf15: Sequentially Ordered Execution in SaltStack talk and slides. Read the full post (123 words, estimated 30 secs reading time)
by Ryan Lane

SaltConf15: Masterless SaltStack at Scale talk and slides

I gave a talk at SaltConf15. It’s about masterless SaltStack, AWS orchestration, Docker management using Salt and other fun things. Enjoy!

Here’s the slides.. Note: though my blog is creative commons licensed, the slides are all rights reserved (sorry!).