Allow 404 pages to be cached

[RealOrangeOne]

Description of Changes Made

If a page returns a 404, chances are it'll still be a 404 in 10 minutes time, or even longer. Therefore, it's probably something which can be cached to reduce system load.

According to RFC2616, 404s should not be cached. However, for our use case, I think it's worth it. The TTL is intentionally shorter than it probably could be, but this could be increased in future.

In Wagtail, a request will always do a database query. Potentially multiple depending on how much of the path does exist. Therefore, missing pages can result in higher than expected usage, and won't be cached by an edge cache. Worse still, because the 404 pages usually shown are fancy HTML versions, they may do queries in themselves (for eg navigation), making 404s more expensive still.

By caching the 404, we reduce the impact on users viewing it in future, especially useful if a site is being crawled, as many frontend caches will normalise URLs before caching (ours sure does).

If a 404 has been cached, and a page is created in its place, Wagtail's existing frontend caching will purge the 404s cache during publishing.

Related reading:

• https://kinsta.com/blog/wordpress-cache/#caching-404-pages
• https://docs.litespeedtech.com/lscache/lscwp/cache/#default-http-status-code-page-ttl
• https://www.drupal.org/project/reuse_cached_404

How to Test

404 pages now have a Cache-Control header which allows them to be cached.

MR Checklist

• Add a description of your pull request and instructions for the reviewer to verify your work.
• If your pull request is for a specific ticket, link to it in the description.
• Stay on point and keep it small so the merge request can be easily reviewed.
• Tests and linting passes.

Unit tests

• Added
• Not required

Documentation

• Updated build docs
• Updated editor guidelines (See https://intranet.torchbox.com/torchbox-com-project-docs for the private link, for Torchbox employees only)
• Updated field spec (See https://intranet.torchbox.com/torchbox-com-project-docs for the private link, for Torchbox employees only)
• Not required

Browser testing

• I have tested in the following browsers and environments (edit the list as required)
  • Latest version of Chrome on mac
  • Latest version of Firefox on mac
  • Latest version of Safari on mac
  • Safari on last two versions of iOS
  • Chrome on last two versions of Android
• Not required

Data protection

• Not relevant
• This adds new sources of PII and documents it and modifies Birdbath processors accordingly

Light and dark mode

• I have tested the changes in both light and dark mode
• The change is not relevant to dark and light mode

Accessibility

• Automated WCAG 2.1 tests pass
• HTML validation passes
• Manual WCAG 2.1 tests completed
• I have tested in a screen reader
• I have tested in high-contrast mode
• Any animations removed for prefers-reduced-motion
• Not required

Sustainability

• Images are optimised and lazy-loading used where appropriate
• SVGs have been optimised
• Performance and transfer of data considered
• If JavaScript is needed alternatives have been considered
• Not required

Pattern library

• The pattern library component for this template displays correctly, and does not break parent templates
• The styleguide is updated if relevant
• Changes are not relevant the pattern library