↓Skip to main content

Aero meets SSM

·5 mins
Programming Tech Clojure Devops Cloud Aws

Aero is a small configuration library from JUXT that extends Clojure's EDN reader with support for tags to interpolate values from environment variables, coerce these values to a specific type, use simple conditionals, and several other useful features. It is also easily extensible through the aero.core/reader multimethod.

We have been using EDN files to configure our Clojure applications at Metail for some time, so aero is a natural fit for us. One of the questions we have been avoiding is what to do with secrets (tokens, passwords, API keys, etc.). The aero README suggests that you hide passwords in a local private file. While that works well for local development (and keeps secrets out of version control), it does not help us orchestrate deployments to cloud servers.

We are already using Consul to store key/value data and considered deploying Vault for the secret storage. We could then have used consul-template to interpolate values from Consul and Vault into our configuration files. While this would have worked, it would have been yet another service to deploy and manage, and it felt a bit heavyweight for our needs.

Last year one of my colleagues evaluated CredStash, a simpler secret store built on Amazon DynamoDB and KMS (Key Management Service). This looked very promising as it is a serverless solution that integrates well with AWS IAM. I was looking to migrate our secrets into CredStash when I stumbled across this blog post, describing an alternative solution using AWS Systems Manager.

AWS Systems Manager (SSM) #

The feature of AWS Systems Manager we are interested in is the Parameter Store:

AWS Systems Manager provides a centralized store to manage your configuration data, whether plain-text data such as database strings or secrets such as passwords. This allows you to separate your secrets and configuration data from your code. Parameters can be tagged and organized into hierarchies, helping you manage parameters more easily. For example, you can use the same parameter name, "db-string", with a different hierarchical path, "dev/db-string” or “prod/db-string", to store different values. Systems Manager is integrated with AWS Key Management Service (KMS), allowing you to automatically encrypt the data you store. You can also control user and resource access to parameters using AWS Identity and Access Management (IAM).

Getting started with the parameter store #

We can use the AWS CLI to add and retrieve parameters. First let's create a KMS key so we can store encrypted parameters:

aws kms create-key --description my-test-key

Make a note of the returned key id - you will need it soon.

To create unencrypted parameters we specify --type String:

aws ssm put-paremeter --name "/adserver/test/db/type" --value "postgresql" --type "String"
aws ssm put-parameter --name "/adserver/test/db/name" --value "adserver_test" --type "String"
aws ssm put-parameter --name "/adserver/test/db/host" --value "localhost" --type "String"
aws ssm put-parameter --name "/adserver/test/db/port" --value "5432" --type "String"
aws ssm put-parameter --name "/adserver/test/db/user" --value "adserver_web" --type "String"

For an encrypted parameter, we specify --type SecureString and the key id:

aws ssm put-parameter --name "/adserver/test/db/password" --value "HbGKSEJIz7FvrFFC" \
  --type "SecureString" --key-id "f9870095-fc23-4b99-b11f-69252e714a1f"

To list all parameters:

aws ssm describe-parameters

To retrieve a parameter:

aws ssm get-parameter --name "/adserver/test/db/user"

Finally, to retrieve an encrypted parameter:

aws ssm get-parameter --name "/adserver/test/db/password" --with-decryption

A Gotcha #

The AWS CLI has an interesting feature: if you pass a URL as a value, the CLI will perform an HTTP GET on the URL to resolve the value - see Github Issue #2507. If you actually want to store a URL using the CLI, the work-around is to use --cli-input-json:

aws ssm put-parameter --cli-input-json '{"Name": "/adserver/test/homepage", "Value": "https://example.com/adserver/", "Type": "String"}'

Back to aero #

Now that we have our parameters in the parameter store, we would like to be able to write our application configuration as:

{:homepage #parameter "/adserver/test/homepage"
 :db-spec {:dbtype   #parameter "/adserver/test/db/type"
           :dbname   #parameter "/adserver/test/db/name"
           :host     #parameter "/adserver/test/db/host"
           :port     #long #parameter "/adserver/test/db/port"
           :user     #parameter "/adserver/test/db/user"
           :password #credential "/adserver/test/db/password"}}

The tag #long is built in to aero and will coerce the string value of the parameter to a long, as expected by clojure.java.jdbc/get-connection. The #parameter and #credential tags don't exist yet; we would like them to lookup a String or SecureString parameter in the parameter store and interpolate this value into the configuration.

We implement these new tags by extending the aero.core/reader multimethod. The excellent amazonica library makes our interactions with the Amazon API straightforward, so our entire implementation looks like:

(ns metail.aero
  (:require [aero.core]
            [amazonica.aws.simplesystemsmanagement :as ssm]))

(defmethod aero.core/reader 'credential
  [{:keys [profile] :as opts} tag value]
  (let [{:keys [parameter]} (ssm/get-parameter {:name value :with-decryption true})]
    (:value parameter)))

(defmethod aero.core/reader 'parameter
  [{:keys [profile] :as opts} tag value]
  (let [{:keys [parameter]} (ssm/get-parameter {:name value :with-decryption false})]
    (:value parameter)))

(def read-config aero.core/read-config)

We have included read-config only for convenience; it allows users to import just one namespace to read a configuration with our new tags.

Access Management #

The AWS parameter store and KMS are fully integrated with AWS IAM, so there is lots of flexibility around how you grant access to keys and parameters. We will create a KMS key for each application and grant the application kms:Decrypt permission on its key through an IAM policy. We will also restrict the parameters it can read to just those it needs through the same policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Statement1",
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "ssm:GetParameters",
        "ssm:GetParameter",
        "ssm:GetParametersByPath"
      ],
      "Resource": [
        "arn:aws:ssm:*:*:parameter/adserver/*",
        "arn:aws:kms:*:*:key/f9870095-fc23-4b99-b11f-69252e714a1f"
      ]
    }
  ]
}

The AWS blog post mentioned earlier shows how to configure a policy for ECS applications. We are currently deloying directly to EC2, so we also create an EC2 instance profile and attach the application role. This profile is attached to the EC2 instance when it is launched and the application can retrieve and decrypt its configuration without further ado.

Conclusion #

Using aero, we can configure our applications from a static file, embedded environment variables, or parameters stored securely and retrieved from AWS SSM. We can switch between these modes simply by changing the configuration file: the application itself does not need to change. This allow us to switch seamlessly between a static configuration during local development and a secure parameter store when deploying to the cloud.

↑