Deployment Checklist

Prior to deploying new changes to your server, there are a number of maintenance activities you must do in preparation.

Are there Couch Views that have been changed?

If you altered any couch views, you will need to preindex the couch views using our preindex_views fab command. We require this to be run on all deploys. If there are no changes, then nothing will happen.

When preindex is run, the view you altered will be made in a new view called <view_name>-tmp. The preindex command will run in the background until the view (or views) finish indexing, and an email alert will go out alerting you of its completion. This can take minutes or hours.

Once the preindex command is done, you can run the deploy fab command, which will rename the new view from <view_name>-tmp, to <view_name>

The reason for this is because if you have existing running code that expects <view_name> to behave in one way, but your new code has new expectations between your code and the view itself, that could make the website unstable.

Did you alter any elastic indices?

A more manual maintenance operation is if you made alterations to the elastic indices. These happen if you change the mapping definitions for our Case, FullCase, XForm, FullXForm, or Exchange mapping files in the corehq repo.

When the mappings are altered, they generate new md5 hashes of the prototype index id. New data will index to those index ids. In order to tell commcarehq to use these new indices, you need to flip the elasticsearch alias name from the old index (another hash), to the new index.

Also, when regenerating your index, you need to preindex much like the views.

Elastic index preindex workflow

• Total time for this should take about 4-5 hours for ALL indices to be reloaded, depending on the database load.
• Find out if any of the index mappings will require reindexing
• Run preindex_views in current established workflow
• Log into your server and cd to the code_root_preindex directory
• Enter virtualenv
• Run fast_reindex commands for indices that need reindexing
  • For the general cases, run python manage.py ptop_fast_reindex_cases --bulk
  • For the general fullcases, run python manage.py ptop_fast_reindex_fullcases --bulk
  • For the general xforms, run python manage.py ptop_fast_reindex_xforms --bulk
  • For the general fullxforms, run python manage.py ptop_fast_reindex_fullxforms --bulk
• Run the reindexing in their own screen/byobu sessions.
• When they are completed you may then do a deploy.
• A fab deploy will run ptop_es_manage --flip_alias on the four pillows (as of this writing) to flip the elastic index aliases where needed. If there are no index changes done, then nothing will happen, but if there are, then the indices will flip.
• specifically, it'll run:
  • python ptop_es_manage --flip_alias --pillow CasePillow
  • python ptop_es_manage --flip_alias --pillow FullCasePillow
  • python ptop_es_manage --flip_alias --pillow FullXFormPillow
  • python ptop_es_manage --flip_alias --pillow XFormPillow
• Once these flip, the index will be a few minutes or hours behind (depending on when and how long the indexing took). The deploy will restart pillowtop using the new pillows/indices, and will catch up since the sequence number will be somewhat behind. Be prepared for that.
• Once sure that the flip is complete, you will need to manually delete the old indices that do not have the aliases applied. This is currently still a manual operation.

Fast reindexing advanced usage

The fast reindexer has a number of options you may use, I recommend running --bulk, but there are a few other options. By default the reindexer will load the entire contents of the view it is configured with and save it to a file on the filesystem. There are options to alter its run in the settings below.

• --noinput - suppress the "are you sure" confirmation
• --chunk <num> - alter the chunk size, defaults to 500
• --bulk - use the bulk loading method. Otherwise do it doc by doc
• --resume - resume a suspended/failed run
• --runfile - required with resume. Load a runfile of the saved view output.
  • The reindexer will output files:
    • ptop_fast_reindex_CommCareCase_2013-03-22-0357_data.json - view data json
    • ptop_fast_reindex_CommCareCase_2013-03-22-0357_seq.txt - database sequence string
  • To run the reindexer to resume, just enter the prefix of the file, so for the use case above, it would be ptop_fast_reindex_CommCareCase_2013-03-22-0357
• --sequence <num> - the integer sequence you want to start from. Defaults to zero, and if you know where it left off, you can enter the number here.