Implementing smart caching of secrets in Azure API Management policies

In a previous blog post, we have seen how to retrieve secrets from Azure Key Vault from an API Management policy. This works great, however, we might start to run into throttling due to the limitations which Key Vault imposes.

This might be due to having an API exposed which we need to call frequently, or because we retrieve secrets from Key Vault in multiple implementations, all of which adds to the restrictions. Luckily, API Management has another policy expression which helps us out here, namely the caching policy.

Scenario

For this blog post, we will continue on the implementation of the previous blog post, where we call Request Bin with an authentication header, where the password is stored in Key Vault. The addition of a cache allows us to re-use the retrieved secret, and can either be built-in to API Management, or we can even use our own cache like Redis. For this scenario, we will use the built-in cache. Consequently, we will only go out and call Key Vault every five minutes to refresh the secret, or whenever we get a 401 (Unauthorized) response from the backend. This last part is important, as the backend service we are calling may decide to refresh it’s secret at any time, which would otherwise result in many failed calls.

Implement caching

The first addition to our policy is storing the secret from Key Vault into our cache. For this purpose, we first need to check if an existing secret is found in the cache. If not, we are going to call Key Vault to retrieve the secret, store it in the cache, and update a variable used later on to call the backend service.

<!-- Get secret from cache -->
<cache-lookup-value key="token" default-value="noToken" variable-name="token" />
<choose>
    <!-- Check if secret was found -->
    <when condition="@((string)context.Variables["token"] == "noToken")">
        <!-- Secret was not found in cache, retrieve secret from Key Vault -->
        <send-request ignore-error="false" timeout="20" response-variable-name="passwordResponse" mode="new">
            <set-url>https://kv-we-retrieve-kv-secret.vault.azure.net/secrets/MySecretValue/?api-version=7.0</set-url>
            <set-method>GET</set-method>
            <authentication-managed-identity resource="https://vault.azure.net" />
        </send-request>
        <!-- Update token variable with retrieved secret -->
        <set-variable name="token" value="@{ var secret = ((IResponse)context.Variables["passwordResponse"]).Body.As<JObject>(); return secret["value"].ToString(); }" />
        <!-- Store retrieved secret in cache -->
        <cache-store-value key="token" value="@((string)context.Variables["token"])" duration="300" />
    </when>
</choose>

Now in the call to the backend we use the token variable, which was either retrieved from the cache or in the call to Key Vault, as input for our authentication header.

<!-- Use token for authorization password -->
<set-header name="Authorization" exists-action="override">        
    <value>@((string)context.Variables["token"])</value>      
</set-header>    

Retry on unauthorized response

The next step is to implement a retry mechanism, which triggers when we receive a 401 Unauthorized response. This would indicate the retrieved secret has expired in the backend, and therefor we should get the new secret from Key Vault. Implementation for this is done by placing the complete policy created above into a retry policy. Moreover, we need to move this to the backend policy scope, as otherwise API Management will never trigger the retry. Seeing how we can only do one action in the backend policy scope, we also need to explicitly call the backend within the retry.

<backend>
    <!-- Retry on failure -->
    <retry condition="@(context.Response.StatusCode == 401)" count="2" interval="1">
        <!-- Logic from previous steps -->
        .
        .
        .
        <forward-request />    
    </retry>
</backend>

What’s more, is that we need to add a check at the start of the retry policy, to see if we are retrying. After all, if this is the case we need to remove the secret from the cache, so that we retrieve the new value from Key Vault. For his we use a variable to keep track if this is the first time we called the API.

<backend>
    <!-- Retry on failure -->
    <retry condition="@(context.Response.StatusCode == 401)" count="2" interval="1">
        <choose>  
            <!-- Check if we are in a retry -->
            <when condition="@(context.Variables.GetValueOrDefault("calledOnce", false))">
                <!-- If so, remove the secret from the cache --> 
                <cache-remove-value key="token" />
            </when>
        </choose>
        <!-- Logic from previous steps -->
        .
        .
        .
        <set-variable name="calledOnce" value="@(true)" />
        <forward-request />    
    </retry>
</backend>

Retain body on retries

Now there is one last step to take. Currently, when we would do a retry for any calls with a message body a failure would occur. This is because on the retry we lose the body of the initial message. To work around this, we start by storing the incoming body in a variable in the inbound policy scope.

<inbound>    
    <choose>
        <when condition="@(context.Request.Body != null)">  
            <!-- Needed, because if we dont use this, retries will result in an error -->  
            <set-variable name="body" value="@(context.Request.Body.As<string>(preserveContent: true))" />
        </when>    
    </choose>    
    <rewrite-uri template="/" copy-unmatched-params="true" />
    <base />  
</inbound>

Now whenever we do a retry, we update the body of the outgoing message with the stored body from this variable.

<backend>
    <!-- Retry on failure -->
    <retry condition="@(context.Response.StatusCode == 401)" count="2" interval="1">
        <choose>  
            <!-- Check if we are in a retry -->
            <when condition="@(context.Variables.GetValueOrDefault("calledOnce", false))">
                <!-- If so, remove the secret from the cache --> 
                <cache-remove-value key="token" />
                <choose>      
                    <when condition="@(context.Request.Body != null)">      
                        <!-- Needed, because if we dont use this, retries will result in an error -->        
                        <set-body>@((string)context.Variables["body"])</set-body>      
                    </when>    
                </choose> 
            </when>
        </choose>
        <!-- Logic from previous steps -->
        .
        .
        .
        <set-variable name="calledOnce" value="@(true)" />
        <forward-request />    
    </retry>
</backend>

And that’s it, we now have implemented the complete flow, passing our secret value in the Authorization header to the backend.

Whenever a request comes in to our API exposed in API Management, we check the cache for the password used in the backend call. If it’s not found, or if we receive a 401 Unauthorized response from the backend, we go to Key Vault to retrieve the secret containing the password, and place it into the cache. The full policy is below for reference.

<policies>
    <inbound>
        <choose>
            <when condition="@(context.Request.Body != null)">
                <!-- Needed, because if we dont use this, retries will result in an error -->
                <set-variable name="body" value="@(context.Request.Body.As<string>(preserveContent: true))" />
            </when>
        </choose>
        <rewrite-uri template="/" copy-unmatched-params="true" />
        <base />
    </inbound>
    <backend>
        <!-- Retry on failure -->
        <retry condition="@(context.Response.StatusCode == 401)" count="2" interval="1">
            <choose>
                <!-- Check if we are in a retry -->
                <when condition="@(context.Variables.GetValueOrDefault("calledOnce", false))">
                    <!-- If so, remove the secret from the cache -->
                    <cache-remove-value key="token" />
                    <choose>
                        <when condition="@(context.Request.Body != null)">
                            <!-- Needed, because if we dont use this, retries will result in an error -->
                            <set-body>@((string)context.Variables["body"])</set-body>
                        </when>
                    </choose>
                </when>
            </choose>
            <!-- Get secret from cache -->
            <cache-lookup-value key="token" default-value="noToken" variable-name="token" />
            <choose>
                <!-- Check if secret was found -->
                <when condition="@((string)context.Variables["token"] == "noToken")">
                    <!-- Secret was not found in cache, retrieve secret from Key Vault -->
                    <send-request ignore-error="false" timeout="20" response-variable-name="passwordResponse" mode="new">
                        <set-url>https://kv-we-retrieve-kv-secret.vault.azure.net/secrets/MySecretValue/?api-version=7.0</set-url>
                        <set-method>GET</set-method>
                        <authentication-managed-identity resource="https://vault.azure.net" />
                    </send-request>
                    <!-- Update token variable with retrieved secret -->
                    <set-variable name="token" value="@{ var secret = ((IResponse)context.Variables["passwordResponse"]).Body.As<JObject>(); return secret["value"].ToString(); }" />
                    <!-- Store retrieved secret in cache -->
                    <cache-store-value key="token" value="@((string)context.Variables["token"])" duration="300" />
                </when>
            </choose>
            <set-backend-service base-url="https://enbjeiqttutuh.x.pipedream.net/" />
            <!-- Use token for authorization password -->
            <set-header name="Authorization" exists-action="override">
                <value>@((string)context.Variables["token"])</value>
            </set-header>
            <set-variable name="calledOnce" value="@(true)" />
            <forward-request />
        </retry>
    </backend>
    <outbound>
        <base />
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

The ARM template to deploy the resources with this post can be found on my GitHub page.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.