-
Notifications
You must be signed in to change notification settings - Fork 46
Help! I'm Trying to Deploy and Something's Going Wrong!
This page is updated periodically. If a wrong direction is provided there will be a mini-post declaring it was updated.
This Wiki page will be updated whenever a deploy operation may require special attention from a system administrator. New directions will be placed at the top, all entries will be given a date. If you are coming to this page and your deployment is seriously out of date you will have to work out all the steps that apply to you. If there is an issue with the directions please post an (issue)[https://github.com/onnela-lab/beiwe-backend/issues?q=is%3Aissue+is%3Aopen]. All actions and directions written here have been debugged on my (Eli's) own deployments, which are always up-to-date.
All migration work was completed roughly around early may.
If you have any existing servers deployed using Ubuntu 23.10 you should use the launch script to terminate servers and launch a new manager. Make sure to pull the repo on the main branch and update your Elastic Beanstalk deployment beforehand using the eb deploy command. (the new manager and worker servers will deploy with the most recent version of the repository.)
Next infrastructure work will be a python 3.11 platform update, which we will do "before October" unless something dire occurs to block it.
It is infrastructure upgrade week! I have been updating all sorts of packages and updating the launch script. This work has so far been successful and will be on its way to the main branch soon.
The data processing servers have been on Ubuntu 18.04, which is now out of support. Unfortunately that LTS timeline did not perfectly overlap with the Ubuntu 24.04 release so it has gotten a little messy. We are waiting on that release, but there were other related platform updates from the several years of built up changes that are included in the short term release 23.10 and that work has been completed.
- tracking issue is here https://github.com/onnela-lab/beiwe-backend/issues/331
- for 24.04 here https://github.com/onnela-lab/beiwe-backend/issues/369
Celery needed a polish, it also has another release mid April that I will probably just roll right in because there shouldn't be any breaking changes.
Update to data processing architecture push notifications - this has not started, not thrilled about it though.
Upgrading to Python 3.11. This is the most complex item, and is described fully in the issue.
Minor updates and clarifications to the Mon 06 Jan 2020 post, removed warning that the page is not up to date.
Due to a bug there may be duplicate files in the ChunkRegistry database table. This causes errors in data processing that are not resolvable without cleaning up the database table and enforcing a uniqueness constraint. You will likely not be able to deploy Beiwe to Elastic Beanstalk servers until your database is cleaned up. (It is possible on large deployments for the database schema update to take so long that EB thinks it has failed.)
-
(Update your copy of the repo to the most recent state of the production or master branch, your preference.)
-
Terminate all manager and worker servers for your cluster, then use the cluster management launch script to create a new manager. This manager will be set up with the most recent version of the codebase (master branch), but it will not attempt to run schema migrations.
-
sshonto the manager server. -
Run the
processing-stopcommand to stop Beiwe from processing active jobs. -
cdinto thebeiwe-backendfolder in the home directory of the ubuntu user. -
The deduplication script can take a while to run, the following command creates a background task that will not be terminated if you exit the ssh session. The
timecommand will simply provide some (poorly formatted) information about how long the operation took to run:nohup time python3 -u scripts/dedupe_chunks_script.py > dedupe.log &You can watch the script execute withtail -f dedupe.log, the task is finished when you see output from thetimecommand. -
After the script finishes you will be able to run the schema migration. Again, this operation takes an amount of time related to the size of your database. Like the previous script I recommend you run it using nohup.
nohup time python3 -u manage.py migrate > migrate.log &. (Note that this operation will appear to stall and do nothing, this is because the compute load is in the database as it enforces uniqueness.) -
run
processing-start. You can watch the celery logs with thelogcand confirm that the system is processing data. Note that the system adds items to the processing queue every 6 minutes.
The Beiwe backend codebase is now officially written in Python 3. There is no further support for Python 2, it is deprecated. There is a legacy branch, python2-legacy, that you can use if you want to stay on the old version. The Python 2 version of the codebase will not receive updates and the project will not accept pull requests for that branch. Please follow the Python 3 conversion wiki page.