Skip to content

Help! I'm Trying to Deploy and Something's Going Wrong!

Jump to bottom Edit New page
Eli Jones edited this page Feb 13, 2020 · 5 revisions

This page is updated periodically. If a wrong direction is provided there will be a mini-post declaring it was updated.

This Wiki page will be updated whenever a deploy operation may require special attention from a system administrator. New directions will be placed at the top, all entries will be given a date. If you are coming to this page and your deployment is seriously out of date you will have to work out all the steps that apply to you. If there is an issue with the directions please post an (issue)[https://github.com/onnela-lab/beiwe-backend/issues?q=is%3Aissue+is%3Aopen]. All actions and directions written here have been debugged on my (Eli's) own deployments, which are always up-to-date.

Thursday 13 Feb 2020

Minor updates and clarifications to the Mon 06 Jan 2020 post, removed warning that the page is not up to date.

Mon 06 Jan 2020

Due to a bug there may be duplicate files in the ChunkRegistry database table. This causes errors in data processing that are not resolvable without cleaning up the database table and enforcing a uniqueness constraint. You will likely not be able to deploy Beiwe to Elastic Beanstalk servers until your database is cleaned up. (It is possible on large deployments for the database schema update to take so long that EB thinks it has failed.)

  1. (Update your copy of the repo to the most recent state of the production or master branch, your preference.)

  2. Terminate all manager and worker servers for your cluster, then use the cluster management launch script to create a new manager. This manager will be set up with the most recent version of the codebase (master branch), but it will not attempt to run schema migrations.

  3. ssh onto the manager server.

  4. Run the processing-stop command to stop Beiwe from processing active jobs.

  5. cd into the beiwe-backend folder in the home directory of the ubuntu user.

  6. The deduplication script can take a while to run, the following command creates a background task that will not be terminated if you exit the ssh session. The time command will simply provide some (poorly formatted) information about how long the operation took to run: nohup time python3 -u scripts/dedupe_chunks_script.py > dedupe.log & You can watch the script execute with tail -f dedupe.log, the task is finished when you see output from the time command.

  7. After the script finishes you will be able to run the schema migration. Again, this operation takes an amount of time related to the size of your database. Like the previous script I recommend you run it using nohup. nohup time python3 -u manage.py migrate > migrate.log &. (Note that this operation will appear to stall and do nothing, this is because the compute load is in the database as it enforces uniqueness.)

  8. run processing-start. You can watch the celery logs with the logc and confirm that the system is processing data. Note that the system adds items to the processing queue every 6 minutes.

Wed 01 Jan 2020

The Beiwe backend codebase is now officially written in Python 3. There is no further support for Python 2, it is deprecated. There is a legacy branch, python2-legacy, that you can use if you want to stay on the old version. The Python 2 version of the codebase will not receive updates and the project will not accept pull requests for that branch. Please follow the Python 3 conversion wiki page.

Add a custom footer
Add a custom sidebar
Clone this wiki locally