Upgrade base image to Debian bookworm

[diox]

Fixes mozilla/addons#9507

Description

This PR updates the base docker image addons-server from Debian buster to bookworm

Context

Debian buster is in long-term support mode, but that will end soon. On top of this, the python docker images for buster are not longer receiving updates. We could switch to bullseye but bookworm is the current stable version and will be supported for longer

Testing

• Run make clean_docker or equivalent to make sure you're rebuilding your deps
• Build the image with make build_docker_image
• If it builds correctly:
  • Stop your containers
  • Tag the newly built image to replace mozilla/addons-server
  • Restart the containers to pick it up
• Test your env by clicking around, running tests etc.
• make shell followed by cat /etc/debian_version should return 12.5

[diox]

Build failed

Weird, it built just fine locally...

[KevinMind]

I ran into this error previously due to the way we are installing the mysql packages.

[KevinMind]

Exception: Can not find valid pkg-config name.
mozilla/addons-server#21 7.197       Specify MYSQLCLIENT_CFLAGS and MYSQLCLIENT_LDFLAGS env vars manually

This was solved on Mac by changing to -default for the mysql packages. Might also have to manually install pkg-config? We should check but this seems like a platform issue.

[diox]

I expected pkg-config to be installed by default, but I guess it's not... I've added it in a27fbf9. I suspect because of the platform it's somehow not needed on my machine but might here, as a fallback of some kind ?

[KevinMind]

Interestingly enough, pkg-config is not installed in either buster or bookworm. I determined this by running

container-diff analyze python:3.10-slim-buster --type=apt

If you change the image to mozilla/addons-server:latest you'll see it's installed.

Tried it for both, and did not find it. So it must be getting installed via one of the other packages we install. You've changed the mysql package target.

[diox]

Since uwsgi is broken locally with this I wanted to look into adding a system check for it (which might prove tricky as I don't really want to launch the command... but anyway) and noticed the smoke test step in the verify docker image looks weird: why is it displaying pip/npm installs ? nvm, we're doing a make update_deps so that makes sense. It just makes the whole thing quite noisy.

[diox]

We'll need to add a test to our verify docker image step that tries to run uwsgi (possibly celery as well), since that's currently failing locally...

pip install -I uwsgi --no-cache fixes it (without --no-cache it doesn't!) but I'm not yet sure what's going on.

[diox]

Our theory is that locally, we're using dependencies from /deps mounted from the local volume and so, if you don't explicitly rebuild deps, you might end up with old ones (on top of having a potentially broken pip cache). We'll need to find a way to address this that doesn't completely negate the benefits of caching...

[KevinMind]

Proposal:

initialize_docker should have a dependent target clean_docker

This should

• remove ./deps/*
• prune docker layer/volume/image cache
• remove ./docker-cache/*

This means if you run initialize_docker or clean_docker you kill your cache and the next run should be totally fresh.

[diox]

I agree, but that wouldn't have been enough here, I wanted to keep my database contents so I wouldn't have run make initialize_docker

[KevinMind]

We could have the docker clean part skip the database? You could also say that the expectation is you try update_docker, if that doesn't work you can try nuking with initialize_docker.

I would actually have expected initialize_docker to run "fresh" by default.

[diox]

Yeah, something along those lines sound good. As a rule anyway, when we upgrade the base image, we should clean deps completely... I am not sure this is a use-case worth automating though.

[KevinMind]

That's why I think it should just be a command you can run arbitrarily. If you run clean_docker it will blow up everything, but for now at least we can not automatically trigger it.

[KevinMind]

@diox something like this? #22050

[diox]

@diox something like this? #22050

Yes, that sounds great, I'll try this. Meanwhile I've added a simple system check that we're able to run uwsgi in this PR: 146fc15

I still need to file an issue for the debug-toolbar bug but it's a regression from #21795, not this PR, so I won't address it here.

[KevinMind]

Can you rebase.

[diox]

Rebased (now contains mozilla/addons#9512 debug-toolbar fix as well)

[KevinMind]

@diox can you include how we should consider verifying this locally? I'm sure you have established a set of steps after going over and over yourself. How do I know this is working?

[KevinMind]

Just to clarify, this check is going to be executed when we run

./manage.py check

Probably worth specifying that in the PR description as it is less than obvious, and I'm not 100% sure it is true even now.

[diox]

I added a docstring in c8c6a3f - the test I had added before should prove it's ran as part of manage.py check, which we trigger in the verify docker image GitHub Action. I've also made sure the check does fail if uwsgi isn't rebuilt.

[diox]

Added instructions. One thing I noticed is that we should also upgrade CircleCI base image to be closer to debian bookworm just in case. Actually, CircleCI is already running cimg/python3.10-node which is based on Ubuntu 22.04, which is the closest thing to debian bookworm we have as far as CircleCI python convenience images go

[KevinMind]

Verified locally.