Skip to content

Help! I'm Trying to Deploy and Something's Going Wrong!

Jump to bottom Edit New page
Eli Jones edited this page Jan 8, 2020 · 5 revisions

This page is not yet finished.

This Wiki page will be updated whenever a deploy operation may require special attention from a system administrator. New directions will be placed at the top, all entries will be given a date. If you are coming to this page and your deployment is seriously out of date you will have to work out all the steps that apply to you. If there is an issue with the directions please post an issue. All actions and directions written here have been debugged on my own deployments.

Mon 06 Jan 2020

(This is a small part of a major update to data processing that will require you to redeploy worker and manager servers anyway.)

Due to a bug there may be duplicate files in the ChunkRegistry database. This causes errors in data processing that are not resolvable without cleaning up the database table and enforcing a uniqueness constraint. You will likely not be able to deploy Beiwe code until your database is cleaned up, and even then it is possible that the database schema update takes so long that Elastic Beanstalk thinks it has failed.

  1. Terminate all manager and worker servers for your cluster, then use the cluster management launch script to create a new manager. This manager will be set up with the most recent version of the codebase, but it will not attempt to run schema migrations.

  2. ssh onto the manager server.

  3. Run the processing-stop command to stop Beiwe from processing active jobs.

  4. cd into the beiwe-backend folder in the home directory of the ubuntu user.

  5. The deduplication script can take a while to run, the following command creates a background task that will not be terminated if you exit the ssh session. The time command will simply provide some (poorly formatted) information about how long the operation took to run: nohup time python3 -u scripts/dedupe_chunks_script.py > dedupe.log & You can watch the script execute with tail -f dedupe.log, the task is finished when you see output from the time command.

  6. After the script finishes you will be able to run the schema migration. Again, this operation takes an amount of time related to the size of your database. Like the previous script I recommend you run it using nohup. nohup time python3 -u manage.py migrate > migrate.log &. (Note that this operation will appear to stall and do nothing, this is because the compute load is in the database enforces uniqueness.)

  7. run processing-start. You can watch the celery logs with the logc and confirm that the system is processing data. Note that the system adds items to the processing queue every 6 minutes.

Wed 01 Jan 2020

The Beiwe backend codebase is now officially written in Python 3. There is no further support for Python 2, it is deprecated. There is a legacy branch, python2-legacy, that you can use if you want to stay on the old version. The Python 2 version of the codebase will not receive updates and the project will not accept pull requests for that branch. Please follow the Python 3 conversion wiki page.

Add a custom footer
Add a custom sidebar
Clone this wiki locally