Update Django to version 3.2

[rtibbles]

Summary

• Updates all Django related dependencies to ones compatible with 3.2
• Updates translation and language code previously vendored from Django for modification
• Updates all url function invocations to re_path
• Updates filterset to filterset_class for DRF viewsets
• Use the new header attribute for response objects to access header values
• Explicitly mark tests that access more than one of the test databases with __all__
• Updates tests that were sending None as a value for a POST - the Django test client no longer allows this
• Changes is_authenticated check from method invocation to property check
• Fixes imports
• Changes function signatures for field methods
• Updates how we serialize and deserialize JSONFields and DateTimeTzFields to match change in Morango - now both serialization and deserialization behaviour are explicitly flagged - parallel to how Django's own serialization framework handles it except for specific reserved types
• Adds the now required --all argument to the makemessages command
• Updates static lib usage
• Adds setting to set the automatic PK field to AutoIncrementingInteger
• Stops deletion of querysets that have had distinct called on them
• Removes python 2 unicode decorator
• Cleans up all swappable dependency cruft from migrations for FacilityUser
• Update Django model fields based on warnings/errors from Django
• Removes bugfixes that are now in Django
• Handle badly formed GET parameters now raising 400s
• Update how we do inheritance for anonymous user and facility user, as anonymous users can no longer be an abstract Model class
• For some bulk operations (channel import, db restore) turn off SQLite referential integrity checks, as they are stricter than PostgreSQL and get verified at statement execution, not transaction commit.
• Update postres and cryptography dependencies
• Removes collections and translations monkey patching that was previously required
• Updates html5lib to not need collections monkey patching

References

Fixes #11726

Reviewer guidance

The urls migration could have been done more selectively by using path instead of re_path in cases where regex is not strictly needed, but that seemed like a task better suited for follow up.

The database attribute of test classes could also have been more selectively applied, but it was easier to use __all__.

Do tests pass?

I didn't update any documentation, as I don't think anything hugely changed.

---

Testing checklist

• Contributor has fully tested the PR manually
• If there are any front-end changes, before/after screenshots are included
• Critical user journeys are covered by Gherkin stories
• Critical and brittle code paths are covered by unit tests

PR process

• PR has the correct target branch and milestone
• PR has 'needs review' or 'work-in-progress' label
• If PR is ready for review, a reviewer has been added. (Don't use 'Assignees')
• If this is an important user-facing change, PR or related issue has a 'changelog' label
• If this includes an internal dependency change, a link to the diff is provided

Reviewer checklist

• Automated test coverage is satisfactory
• PR is fully functional
• PR has been tested for accessibility regressions
• External dependency files were updated if necessary (yarn and pip)
• Documentation is updated
• Contributor is in AUTHORS.md

[github-actions]

Build Artifacts

Asset type	Download link
PEX file	kolibri-.pex
Windows Installer (EXE)	kolibri-0.16.2b1.dev0+git.1034.g6f7db014-windows-setup-unsigned.exe
Debian Package	kolibri_0.16.2b1.dev0+git.1034.g6f7db014-0ubuntu1_all.deb
Mac Installer (DMG)	kolibri-0.16.2b1.dev0+git.1034.g6f7db014-0.4.1.dmg
Android Package (APK)	kolibri-0.16.2b1.dev0+git.1034.g6f7db014-0.1.2-debug.apk
TAR file	kolibri-0.16.2b1.dev0+git.1034.g6f7db014.tar.gz
WHL file	kolibri-0.16.2b1.dev0+git.1034.g6f7db014-py2.py3-none-any.whl

[pcenov]

Hi @rtibbles, after a clean installation of the .deb build Kolibri is not loading the setup wizard. Logs and DB:
logs.zip

[rtibbles]

Ah, thanks @pcenov! Looks like none of the frontend files are being properly loaded - guessing Django is bypassing our previous setup somehow.

[rtibbles]

Latest commit should have fixed this - updated assets will be along in a little while!

[pcenov]

Hi @rtibbles now the Setup Wizard is working correctly but it's not possible import a channel in any of the supported ways:

2024-03-11_12-26-58.mp4

Logs and DB: logs.zip

[rtibbles]

Who knew that changing everything would be so difficult?! :)

[rtibbles]

Looks like there's some more issues with how we parse arguments for the import channel command (and probably the import content command too).

[rtibbles]

@pcenov this should now be fixed. I also looked for other possible instances of the same error. If you could also test the facility import, single user import + syncing, on my own setup + merge to single user syncing, and facility syncing that should cover the places that I updated.

[pcenov]

Hi @rtibbles, I found two additional issues while regression testing everything:

1. The Android app can be installed but closes immediately after I launch it. Logs: android-logs.zip
2. When migrating a user, the process gets stuck at "Remotely integrating data". If I sign out and sign in again with the same user I can see that the user is actually migrated. Logs: migrate-user-logs.zip

2024-03-13_17-56-28.mp4

[rtibbles]

Hi @pcenov - the android app kolibri log is exceptionally short. Could you rerun the app and grab the logcat output? That might give me some more clues.

[rtibbles]

Some very interesting bugs in the other log, can take a closer look at them!

[pcenov]

Hi @rtibbles here's the logcat log: logcat.txt

[rtibbles]

Thank you @pcenov - hah, I think this might be a result of our not having merged 0.16.x into develop recently enough rather than anything I've done here, so I'll get that sorted too. Thank you!

[rtibbles]

Oh, on further investigation, that doesn't seem to be the issue - it just looks very similar to an error we had late on in the 0.16.0 release process.

[rtibbles]

Hi @pcenov - I think I've addressed all the errors that I saw in the logs.

[pcenov]

Hi @rtibbles - I confirm that the Android app installs and runs correctly now. I'm still testing the critical workflows with all installers and will let you know if I find anything else that's not working.

[pcenov]

Hi @rtibbles the only new issues I was able to identify are for the Mac app. There I am getting a server error at Device > Channels > Select a source which remains in a 'Loading connections' state and it's not possible to import resources.

Also no libraries are being displayed in the 'Other libraries' section of the Library page:

logs.zip