Improving cache hit ratio

[ddonahue99]

My company recently deployed the serverless image handler, and it was a breeze - nice work! One thing we've noticed that has been a little surprising is a lower than expected CloudFront cache hit ratio, and we'd love to be able to get the Lambda costs down. My assumption is that the serverless image handler is caching at each CloudFront edge location, so for a given image requested from several places around the globe, it will need to hit the lambda multiple times. Over time, those cached items will expire and will need to be re-hydrated again. Is that correct?

Assuming that's what's going on, a couple options come to mind for optimizing the hit ratio:

1. Cache the converted images in S3, rather than relying solely on the CloudFront cache. Storage costs would be higher, but it would need to hit the lambda exactly once for a given set of image parameters. This would obviously require some fundamental changes to the serverless-image-handler.

2. For a lighter approach, would CloudFront Origin Shield solve this problem? Would need to crunch the numbers to evaluate cost implications, but it seems like it exists for this sort of use case.

Thanks in advance for any guidance, and please let me know if there are any other options I am not considering.

[gattasrikanth]

Thanks for using Serverless Image Handler Solution.
I have added this to our backlog items list and our dev team will look into possible solutions to optimize.

[Buthrakaur]

Hi @ddonahue99 , do you have any experience with the CloudFront Origin Shield already? I'm just thinking about using it too to at least somehow limit the Lambda execution count/time..

[ddonahue99]

Hi @Buthrakaur - Since posting this, I've made a few changes that have greatly improved the hit ratio, including enabling Origin Shield (which resulted in a modest improvement).

The more notable impact, however, was modifying the CloudFront cache settings. I bumped the TTL up to the max (1 year) and changed the cache key to not include the origin and accept headers. From what I could tell, the accept header is part of the cache key by default for the AUTO_WEBP setting, which makes sense, because depending on the client, the response could be webp or jpeg or whatever other fallback you specify. If you are not using AUTO_WEBP, the response will always be the same, so it doesn't make sense to have roughly one cache entry per major browser:

Example of how accept headers vary by browser:

firefox = image/webp,*/*
safari = image/webp,image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5
chrome = image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8

With all of these changes, my application was hovering around a 70-75% hit ratio and is now closer to 96%.

Ultimately, the better solution for optimizing the hit ratio would be to permanently cache the output in S3. I'd still love to see that as a built-in option to this template. 🙏

[fvsnippets]

Hi!

Noticed another possible improvement:
Current cache policy (the one provided by current version of serverless-image-handler), enables gzip compression which in turn adds the "accept-encoding" header to the cache key. But origin (resizing lambda) won't use it (am I missing something?) which makes sense because we are working with already compressed image-formats.
Notice that cloudfront enables auto compression only on image/svg+xml (see this) which also makes sense.
So, with current cache policy, two almost "equivalent" requests (from cloudfront's cache key perspective), except for the accept-encoding: gzip header presence, will generate different entries on the cache.

Please read this: "Cache Hit Ratio - Remove Accept-Encoding header when compression is not needed".

[fvsnippets]

Hi @ddonahue99 , I am seeing that TTL configuration is already one year when image on s3 doesn't provide one, and when s3 file provides a TTL then it will use that one. See this.

Cloudfront will honor cache-control header from origin when it provides one.

Maybe... am I missing something? Please give us some details (I am currently working on improve hit ratio too).

[fvsnippets]

Notice: making modifications to allow enabling Origin Shield optionally (enabled by parameter) on the solution, is a little complicate on the current CDK definition (or at least I can't figure out a simple way). But a solution's user could modify the provided template.yaml to add it. It's as simple as:

  BackEndImageHandlerCloudFrontApiGatewayLambdaCloudFrontToApiGatewayCloudFrontDistribution03AA31B2:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        ...
        Origins:
          - CustomOriginConfig:
            ...
+           OriginShield:
+             Enabled: true
+             OriginShieldRegion: us-west-2
            ...

As a simple workaround, I would suggest to the mantainers to add it to the documentation.

[fvsnippets]

One other possible optimization [0].
The following applies when we have NOT enabled a Default Fallback Image.

Thumbor requests without id [1] (such as "/fit-in/120x120/") are forwarded to the origin.
The Lambda backend receives the request and processes it with a 500 error including a message "Expected uri parameter to have length >= 1, but found "" for params.Key". In the first place, I would like to say that I think this an erroneous behavior because it isn't an error from the processing lambda but from the requester. I think that it should be handled as a 400 or arguably a 404 status code [2]. But that's a topic for another issue.

The important thing here is that we could avoid making a request to the origin, just using a CloudFront Function (o a Lambda@Edge extension) matching the incorrect path.

I'll show an example:

+  BackEndCfnFunctionFB18E3BF:
+    Type: AWS::CloudFront::Function
+    Properties:
+      Name: fastNotFoundResponseFunction
+      AutoPublish: true
+      FunctionCode:
+        Fn::If:
+          - CommonResourcesEnableCorsConditionA0615348
+          - Fn::Join:
+              - ""
+              - - |-
+                  function handler(event) {
+                    // Notice: cannot modify body on fast responses from Cloudfront Functions. But we should be ok with that.
+
+                    if (event.request.method == 'GET') {
+                      var fastNotFoundPathsRegex = new RegExp('^/fit-in/[0-9]+x[0-9]+/?$');
+
+                      if (fastNotFoundPathsRegex.test(event.request.uri)) {
+                        return {
+                          statusCode: 404,
+                          statusDescription: 'Not Found',
+                          headers: {
+                            'content-type': { value: 'application/json' },
+                            'access-control-allow-methods': { value: 'GET' },
+                            'access-control-allow-headers': { value: 'Content-Type, Authorization' },
+                            'access-control-allow-credentials': { value: 'true' },
+                            'access-control-allow-origin': { value: '
+                - Ref: CorsOriginParameter
+                - |-
+                  ' }
+                          }
+                        };
+                      }
+                    }
+
+                    return event.request;
+                  }
+          - |-
+            function handler(event) {
+              // Notice: cannot modify body on fast responses from Cloudfront Functions. But we should be ok with that.
+
+              if (event.request.method == 'GET') {
+                var fastNotFoundPathsRegex = new RegExp('^/fit-in/[0-9]+x[0-9]+/?$');
+
+                if (fastNotFoundPathsRegex.test(event.request.uri)) {
+                  return {
+                    statusCode: 404,
+                    statusDescription: 'Not Found',
+                    headers: {
+                      'content-type': { value: 'application/json' },
+                      'access-control-allow-methods': { value: 'GET' },
+                      'access-control-allow-headers': { value: 'Content-Type, Authorization' },
+                      'access-control-allow-credentials': { value: 'true' }
+                    }
+                  };
+                }
+              }
+
+              return event.request;
+            }
+      FunctionConfig:
+        Comment: Returns not-found responses to some already know to be existent but frequently requested paths.
+        Runtime: cloudfront-js-1.0
      ...
  BackEndImageHandlerCloudFrontApiGatewayLambdaCloudFrontToApiGatewayCloudFrontDistribution03AA31B2:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        ...
        DefaultCacheBehavior:
          ...
+          FunctionAssociations:
+            - EventType: viewer-request
+              FunctionARN:
+                Fn::GetAtt:
+                  - BackEndCfnFunctionFB18E3BF
+                  - FunctionARN
          ...

Notice:

• CloudFront Functions cannot include a response body (only headers and status code are allowed). Lambda@Edge allows that (but I am staying with CloudFront Functions for simplicity and speed).
• returning 404 as previously discussed.

Of course, this approach can also be applied to some other previously know to be always invalid/not existent (but frequently requested) paths.

As with optionally enabling Origin Shield (see my previous message), enabling this code based on EnableDefaultFallbackImageParameter value is a little complicated on the current CDK definition (or at least I can't figure out a simple way). But readers can make a custom modification of the base template.yaml.

[0] = being strict, the proposal here is not a cache optimization: CloudFront Functions are run before using cache. But might avoid making requests to the origin, which, in the end, achieves the same.
[1] = and maybe no-Thumbor requests too; I don't know because I don't use them, and I know almost nothing about them.
[2] = notice that current CloudFront's configuration caches 500 status code responses for ten minutes, whereas it caches 400/404 status code responses for only ten seconds.

[fvsnippets]

As a picture is worth a thousand words, these are my results (invocations on backend lambda) after applying all these things at the same time; sorry that was what I did, so I cannot show them one at a time.
That is:

• removing "accept" header from cache key PLUS
• disabling gzip compression (removing accept-encoding from cache key) PLUS
• enabling Origin Shield PLUS
• using CloudFront Functions to answer incorrect but yet frequently requested paths (I am not using "Default Fallback Image"; my consumers are resolving that, but that's only my case).

Initial peak is attributable to old caches invalidation due to cache key conformation being modified.

---

But @ddonahue99 proposal of caching converted images on s3 would be a very important improvement, because CloudFront's caches (I understand that this applies to POPs, Regional and Shield caches) will discard less popular objects (please read this)

[ddonahue99]

@fvsnippets It looks like you made a really meaningful dent, nice work and thank you for sharing all of your findings! I'm going to have to investigate the Accept-Encoding and fallback image tweaks on my end as well. It's been a while since we've revisited the configuration, but our hit ratio is still hovering around the low-to-mid 90s, so there's some more room for improvement.

If the AWS team is open to allowing for permanent caching in S3, I still agree that would have the biggest impact over the long-term. This solution is not the most efficient as-is for performance/cost at scale.

[asgerjensen]

maybe i'm wrong, but the main issue with doing straight serving from s3 in cloudfront is how to map the cloudfront cache key to a filename, especially when using things like AUTO_WEBP (and especially once AUTO_WEBP also does AUTO_AVIF ;)) without adding the runtime cost of another lambda edge call (time, and money).

I suppose it could be dealt with by the image handler, by having it check a CACHE_BUCKET once it has fully resolved all parameters, and immediatly prior to actually loading the image from the SOURCE BUCKET and performing operation.

if present, return it, as if it had been through the entire process, and if not, proceed and store the output to the CACHE_BUCKET.

it does mean, it will not do CDN => CACHE_S3? => API-GW, but instead CDN => API-GW => CACHE_S3, so you wont save on the api-gw calls, but you /will/ save on customer wait time for items that are already processed once.

[fvsnippets]

Maybe it could be enabled only under certain circumstances (AUTO_WEBP not enabled, etc) and only for certain paths (e.g. Thumbor resize URLs). I understand that CloudFront allows the latter by using origin groups (but I haven't read enough/have experience on that topic to tell for sure).