Fix caching issues for remote content API calls

[rtibbles]

Summary

• Fixes issue whereby non-200 responses would get cached
• Fixes content API caching to bypass Django caching middleware, as bespoke caching mechanisms are being used instead
• Fixes public content API endpoints to use matching cache headers to the same endpoints on Studio
• Fixes Etag checking on access to remote libraries by keeping a cache of Etags by baseurl to allow quick 304 responses without having to go to the remote

References

Fixes #11457

Reviewer guidance

Load e.g. http://127.0.0.1:8000/api/public/v2/contentnode_tree/d6e3b856125f5e6aa5fb40c8b112d5e9/?format=json and check that refreshes result in a 304 immediately

Likewise for access to remote URLs, such as http://localhost:8000/api/content/contentnode_tree/d6e3b856125f5e6aa5fb40c8b112d5e9/?include_coach_content=true&baseurl=https%3A%2F%2Fstudio.learningequality.org&format=json

Also follow the steps outlined in the issue to ensure that 410s are not being cached.

---

Testing checklist

• Contributor has fully tested the PR manually
• If there are any front-end changes, before/after screenshots are included
• Critical user journeys are covered by Gherkin stories
• Critical and brittle code paths are covered by unit tests

PR process

• PR has the correct target branch and milestone
• PR has 'needs review' or 'work-in-progress' label
• If PR is ready for review, a reviewer has been added. (Don't use 'Assignees')
• If this is an important user-facing change, PR or related issue has a 'changelog' label
• If this includes an internal dependency change, a link to the diff is provided

Reviewer checklist

• Automated test coverage is satisfactory
• PR is fully functional
• PR has been tested for accessibility regressions
• External dependency files were updated if necessary (yarn and pip)
• Documentation is updated
• Contributor is in AUTHORS.md

[akolson]

It looks like I am still able to replicate the 410s by following #11457 replication steps. Maybe rebasing to include #11459 (review) changes could prove beneficial?

410s.mov

[github-actions]

Build Artifacts

Asset type	Download link
PEX file	kolibri-0.16.0b6.dev0_git.289.g1092b499.pex
Windows Installer (EXE)	kolibri-0.16.0b6.dev0+git.289.g1092b499-unsigned.exe
Debian Package	kolibri_0.16.0b6.dev0+git.289.g1092b499-0ubuntu1_all.deb
Mac Installer (DMG)	kolibri-0.16.0b6.dev0+git.289.g1092b499-0.3.0.dmg
Android Package (APK)	kolibri-0.16.0b6.dev0+git.289.g1092b499-0.1.0-debug.apk
TAR file	kolibri-0.16.0b6.dev0+git.289.g1092b499.tar.gz
WHL file	kolibri-0.16.0b6.dev0+git.289.g1092b499-py2.py3-none-any.whl

[rtibbles]

Ugh - no, this seems to be that in spite of refusing to use disk cache for other requests, browsers are very happy to use the disk cache for the 410 response! I'll patch the response headers in this case to prevent caching.

[rtibbles]

@akolson I've made one more update to ensure that we add never cache headers to non-200 responses as well, so that browsers are never tempted to cache them!

[akolson]

Changes look correct and also seem to fix the issue reported. Tagging @radinamatic @pcenov for more manual QA. Thanks @rtibbles

[pcenov]

LGTM as well, no caching issues observed while manually testing.