Lazy Paging with the Cognitect AWS API

The aws-api library from Cognitect provides programmatic access to AWS resources from Clojure. Many of the endpoints in the AWS REST API provide paged responses using a continuation marker. For example, ListPolicies in the IAM API returns IsTruncated and Marker in a response, and accepts an optional Marker in the request parameters. We can query these operations from Clojure:

(require '[cognitect.aws.client.api :as aws])

(def iam (aws/client {:api :iam}))

(aws/doc iam :ListPolicies)

This shows us the request parameters:

{:Scope [:one-of ["All" "AWS" "Local"]],
 :OnlyAttached boolean,
 :PathPrefix string,
 :PolicyUsageFilter
 [:one-of ["PermissionsPolicy" "PermissionsBoundary"]],
 :Marker string,
 :MaxItems integer}

and the response parameters:

{:Policies
 [:seq-of
  {:PermissionsBoundaryUsageCount integer,
   :Path string,
   :Tags [:seq-of {:Key string, :Value string}],
   :CreateDate timestamp,
   :PolicyName string,
   :AttachmentCount integer,
   :DefaultVersionId string,
   :IsAttachable boolean,
   :Arn string,
   :Description string,
   :PolicyId string,
   :UpdateDate timestamp}],
 :IsTruncated boolean,
 :Marker string}

To list all policies, we have to call the API repeatedly with the returned :Marker until :IsTruncated returns false. We can do this lazily, a page at a time, like so:

(defn list-policies
  [iam request]
  (let [{:keys [Policies IsTruncated Marker]} (aws/invoke iam {:op :ListPolicies :request request})]
    (if IsTruncated
      (lazy-cat Policies (list-policies iam (assoc request :Marker Marker)))
      Policies)))

If we are working with lots of different resource types, it will be useful to generalize this lazy paging to other list operations and to other AWS APIs. Within an API the keys for the results will change, for example if we invoke the IAM :ListUsers operation the results will be under the :Users key. If we call a different AWS API, the name of the continuation markers might change. For example, the S3 :ListObjectsV2 operation returns :NextContinuationToken in the response, and expects :ContinuationToken in the request.

The following function generalizes the list-policies function to handle paged responses from other endpoints:

(defn lazy-page
  [{:keys [op results is-truncated marker next-marker]
    :or {results :Contents is-truncated :IsTruncated marker :Marker next-marker :Marker}
    :as options}
   client request]
  (let [response (aws/invoke client {:op op :request request})]
    (if (is-truncated response)
      (lazy-cat (results response)
                (lazy-page options client (assoc request marker (next-marker response))))
      (results response))))

Now to lazily page through a list of policies we could write:

(lazy-page {:op :ListPolicies :results :Policies} iam nil)

We could also rewrite our original list-policies function to use the helper:

(def list-policies (partial lazy-page {:op :ListPolicies :results :Policies}))

For the S3 list objects operation, which uses different names for the page markers, we would write:

(def list-objects (partial lazy-page {:op :ListObjectsV2 
                                      :results :Contents 
                                      :marker :ContinuationToken 
                                      :next-marker :NextContinuationToken}))

Making the request an optional argument #

It's a bit annoying to have to pass a nil argument when we don't want to pass additional request parameters. We can address this by implementing another arity of the function:

(defn lazy-page
  ([options client]
   (lazy-page options client nil))
  ([{:keys [op results is-truncated marker next-marker]
     :or {results :Contents is-truncated :IsTruncated marker :Marker next-marker :Marker}
     :as options}
    client request]
   (let [response (aws/invoke client {:op op :request request})]
     (if (is-truncated response)
       (lazy-cat (results response)
                 (lazy-page options client (assoc request marker (next-marker response))))
       (results response)))))

Error Handling #

None of the functions we have written so far do any error handling: if the response contains an error, it is silently ignored. The aws-api documentation tells us:

If AWS indicates failure with an HTTP status code >= 400, the map will include a :cognitect.anomalies/category key, so you can check for the absence/presence of that key to determine success/failure.

Let's add some error handling to give the final version of our paging helper:

(defn lazy-page
  ([options client]
   (lazy-page options client nil))
  ([{:keys [op results is-truncated marker next-marker]
     :or {results :Contents is-truncated :IsTruncated marker :Marker next-marker :Marker}
     :as options}
    client request]
   (let [response (aws/invoke client {:op op :request request})]
     (when (contains? response :cognitect.anomalies/category)
       (throw (ex-info "aws/invoke error" 
              {:op op :request request :response response})))
     (if (is-truncated response)
       (lazy-cat (results response)
                 (lazy-page options client (assoc request marker (next-marker response))))
       (results response)))))