move deploy repo assets to webapp-public

[PeterDSteinberg]

(This is a fix of PR #552 that I just closed)

This is a work in progress to move the assets of the deploy private repo to this webapp-public one. Goal is to address #543 , but the work may be done in several PRs. It aims to simplify and better test the interaction of repos, mainly the Django app calling out to Tax-Calculator, B-Tax, OG-USA models

What do you think of the folder name taxbrain_server I made? That's will be a Python standalone 1-level package, e.g. from taxbrain_server.celery_tasks import *?
cc @brittainhard @MattHJensen @martinholmer @GoFroggyRun @jdebacker @DoraSzasz

[martinholmer]

@PeterDSteinberg, Why in #553 is deploy/fab/dropq_environment.yml hardwired to use python 2.7.9 when that is out-of-date on the Anaconda download page?

[PeterDSteinberg]

@martinholmer That's a good point - we should remove that python hard coding of 2.7.9. It has been that way for a while in the deploy repo.

[martinholmer]

@PeterDSteinberg said in #553:

we should remove that python hard coding of 2.7.9. It has been that way for a while in the deploy repo.

I'm glad you see it that way. We're making an effort in Tax-Calculator pull request 1351 to be using the latest versions of Python and the rest of the packages Tax-Calculator depends on.

[PeterDSteinberg]

TODO:

• Address the Python versioning (see @martinholmer 's comment above)
• Adjust as needed relative to file moves, such as creation of the taxbrain_server pkg
• Make a setup.py file in ./deploy
• Make a conda.recipe folder in ./deploy - EDIT: instead of conda.recipe I made a install_taxbrain_server.sh script
• Deploy and redeploy the test app several times with this PR
• IMPORTANT: Add some usage check controls to reset_server.sh to ensure it is not run outside a linux server setting. Do not run reset_server.sh on your laptop: it is not designed for that - it tears down aei_dropq conda env among other things.
• Change webapp-public's testing to take advantage of this new set up:
  • Make a test harness to simulate a request from Django, but actually have it call a taxbrain_server function directly rather than wait for it to be done by a flask server
  • Change webapp-public's Travis config so that this new taxbrain_server package is installed and tested along with webapp-public PRs / pushes, and make sure the testing is inclusive of interactions between taxbrain_server and the Django app.
• Make sure taxbrain_server gets installed on the EC2 back end workers
  • Adjustments to reset_server.sh ?
  • Change bash scripts / salt commands for starting flask and celery?
  • Adjustments to redeploy.py?
• Provide new documentation to update what was the deploy documentation for new practices

Does that seem like an adequate TODO list, @brittainhard @MattHJensen @martinholmer @jdebacker @DoraSzasz @GoFroggyRun

[PeterDSteinberg]

Current usage of the assets from preceding commits that created the taxbrain_server Python package in the ./deploy folder of this repo:

Install taxbrain_server

Installing taxbrain_server for local use (this is not the exact installation pathway done on EC2):

git clone http://github.com/OpenSourcePolicyCenter/webapp-public
cd webapp-public/deploy
. install_taxbrain_server.sh

Testing all services locally

Do each of the commands below in separate terminals - they replace the run_celery.sh and run_flask_server.sh scripts we deploy repo has used until now.

Run celery:

taxbrain-celery-worker

Run flask:

taxbrain-flask-worker

Run redis

redis-server

Run Django

See webapp-public README for running Django app

[PeterDSteinberg]

TODO:

• Make a installation checker for dropq workers vs heroku: a bash script that checks conda list on front end and back end to make sure they are on same versions of stack. It probably needs to do ssh and heroku config checks to see taxcalc, btax, ogusa, pandas, numpy, python versions and make sure they are consistent.