Upgrade API to use Elasticsearch 5.6

[lbeaufort]

Cloud.gov now has support for Elasticsearch version 5.6 (wohoo!) more info here, which may improve some of the performance issues we've been having with our current version (2.4).

Also, cloud.gov may end support for 2.4 in the medium-term future.

• pre-update load testing baseline - dev and local. [Saved here] (https://drive.google.com/drive/folders/1F4lFVVC6gFfN7rwhfNl41WvhUBkksimp?ths=true)
• Redo local testing after loading all AO's and MURs? Restore snapshot for consistency?
• How do we deal with backups and/or reloading archived MURs onto new version? Looks like we can restore a 2.X snapshot to 5.x - https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html
• Test elasticsearch v. 5.6 locally -loading docs, reindexing, load test, check out the front-end, API, etc
• update requirements.txt and manual deploy. Update service? https://cloud.gov/docs/services/elasticsearch56/
• manual testing of dev - front-end, API, loading docs, reindexing, etc
• load testing dev - confirm no performance hit and possible improvement
• Update README.md

Remaining items

• delete 2.4 service instance on dev once the PR is merged: cf delete-service SERVICE_INSTANCE
  Before release cut on 5/22:
• Create 5.6 service instances in stage: cf create-service elasticsearch56 medium-ha fec-api-search56
• use a headless app to bind to the 5.6 instances
• restore from snapshot so we will have no downtime
  After release cut on 5/22:
• Reload legal docs to be sure everything looks good
• delete the 2.4 service instance
  Before deployment on 5/29:
• Repeat above steps for the prod space
  After deployment on 5/29:
• delete the 2.4 service instance

From cloud.gov: "Elasticsearch version 5.6 is now available. This includes performance improvements and new features. We recommend upgrading from version 2.4 to 5.6, but we don’t have near-term plans to end support for 2.4. "

[lbeaufort]

@fec-jli requested that we pair on this during the first week of 5.5 before she goes on vacation This got moved back to 5.4

[pkfec]

Create Elastic Search 5.6 service instances in Production: cf create-service elasticsearch56 medium-ha fec-api-search56 during the release on May 30th

Ran the following command to bind this new service to the api : cf bind-service api fec-api-search56

[lbeaufort]

No remaining items for this ticket. Thanks for all your hard work! @vrajmohan @pkfec @fec-jli

[lbeaufort]

@vrajmohan I confirmed doc counts on prod here: https://docs.google.com/spreadsheets/d/1Ksb3mjwsTKcdAwkRkP8n3e6KrNmoZor_WP1ipDQmibo/edit#gid=0

Archived MURs 4445
Current Murs 2160
advisory_opionions 1989
citations 531
regs 529
statutes 57

Total All 9711