API Umbrella upgrade for production site

[GUI]

There's a substantial update to API Umbrella in the pipeline. It will finally get a lot of the pending issues pushed live on api.data.gov (including #56, #55, #121, #112, #121, #122). This issue is to coordinate what needs to happen in order to get that live.

The basic plan so far:

• Spin up the staging servers and make sure their code base is updated.
• Sync the production mongodb database back to staging (includes users).
• Sync the production elasticsearch database back to staging (includes analytics).
• Expand the staging db server disk space so there's enough room for the upgrade.
• Expand the production db server disk space so there's enough room for the upgrade.
• Build new chef scripts to install the updated API Umbrella stack from RPMs.
• Run chef script to install the new API Umbrella software and install config files, but do not start the new services automatically (we'll manually stop old stuff and start new stuff to try to minimize downtime)
• Upgrade Elasticsearch (see cluster restart upgrade process):
  • Disable shard reallocation
  • Stop all old elasticsearch servers
  • Copy data into new database directory location
  • Start new elasticsearch servers
  • Re-enable shard reallocation
• Upgrade MongoDB (see 2.4 to 2.6):
  • Stop one mongo server. Ensure another becomes primary.
  • Copy data into new database directory location
  • Start new mongo server.
  • Repeat process for each replica, one at a time.
• Upgrade API Umbrella Router/Gatekeeper/Web
  • Remove server from load balancer
  • Shutdown the old service. This includes these services:
    • nginx
    • redis
    • supervisor
    • varnish, varnishlog, varnishncsa
  • Startup new service
  • Test and add back to load balancer if everything looks good
• Adjust chef scripts so that the new API Umbrella services automatically start on each server

[GUI]

It's alive!