Move deploy

.buildpacks

@@ -1 +1 @@
-https://github.com/talumbau/conda-buildpack
+https://github.com/PeterDSteinberg/conda-buildpack

deploy/README.md

@@ -0,0 +1,62 @@
+# deploy
+
+How to run webapp-public locally.
+* Clone this `deploy` repo, then `cd` into the `fab` directory and run: `conda env create --file dropq_environment.yml --name aei_dropq && source activate aei_dropq`
+* Follow the instructions for [the `taxpuf` package installation](https://github.com/OpenSourcePolicyCenter/taxpuf) and run `write-latest-taxpuf` from the `deploy` repo clone top dir.
+* Then install Tax-Calculator, OG-USA, B-Tax, and webapp-public into the `aei_drop` environment using conda packages or installations from source.  An example is:
+```
+conda install -c ospc taxcalc ogusa btax # install the latest from "main" ospc channel
+```
+* Copy `run_flask_server.sh` to `run_flask_server_local.sh` so you can edit it.  *Do not commit changes* to `run_flask_server.sh` unless the changes are specific to the deployed EC2 box settings.
+* Copy `run_celery.sh` to `run_celery_local.sh` so you can edit it.  Same comment about changes to `run_celery.sh`.
+* Edit `run_flask_server_local.sh` and `run_celery_local.sh` to define the environment variables in there.  Here's an example of `run_flask_server_local.sh` for me:
+```
+#!/usr/bin/env bash
+export C=/Users/peter/Documents
+BTAX_CUR_DIR='$C/AEI/B-Tax/btax/' TAX_ESTIMATE_PATH="$C/AEI/OG-USA/Python" OGUSA_PATH="$C/AEI/OG-USA/Python/" REDISGREEN_URL="redis://localhost:6379" python flask_server.py
+```
+In the example above, I use the environment variable `C` for where I clone repos, then I create the `BTAX_CUR_DIR` variable as the `btax` directory in the B-Tax clone, `TAX_ESTIMATE_PATH` for OG-USA's `Python` dir and `OGUSA_PATH` to that `Python` dir as well.  `REDISGREEN_URL` tells the server where to look for `redis` to be running.  The env variables are prepended to `python flask_server.py` so the variables apply in that process but are not set in the calling environment.
+
+Then in one terminal, run:
+```
+. run_flask_server_local.sh
+```
+Next edit `run_celery_local.sh` to something like this:
+```
+#!/usr/bin/env bash
+export C=/Users/peter/Documents
+BTAX_CUR_DIR='$C/AEI/btax_new/btax/' TAX_ESTIMATE_PATH="$C/AEI/OG-USA/Python" OGUSA_PATH="$C/AEI/OG-USA/Python/" REDISGREEN_URL="redis://localhost:6379" CELERYD_PREFETCH_MULTIPLIER=1 celery -A celery_tasks worker --concurrency=1 -P eventlet -l info
+```
+Basically just change the env variables that are related to where you cloned things as done for
+
+`. run_flask_server_local.sh`
+
+In a separate terminal tab or window, run:
+
+```
+. run_celery_local.sh
+```
+
+Next, start redis:
+```
+redis-server
+```
+
+Finally, [start your webapp-public django app](https://github.com/OpenSourcePolicyCenter/webapp-public/), following the instructions there for `migrate.py` and `runserver.py`.  Note that when running webapp-public's Django app locally, you can use environment variables to set where each type of compute job is being sent:
+
+ * `TAXCALC_WORKERS`: Where to send /taxbrain jobs
+ * `BTAX_WORKERS`: Where to send /ccc jobs
+ * `OGUSA_WORKERS`: Where to send dynamic models
+
+An example of running webapp-public locally while using an EC2 box as a Tax-Calculator, OG-USA, and B-Tax worker:
+
+```
+TAXCALC_WORKERS=54.164.155.6 BTAX_WORKERS=54.164.155.6 OGUSA_WORKERS=54.164.155.6 python manage.py runserver
+```
+More than one IP address can be given for each of the `*_WORKER` env variables if separated by commas.
+
+If you want to test your local webapp-public django app versus an EC2 box, then prepend `DROPQ_WORKERS`, `BTAX_WORKERS`, or `OGUSA_WORKERS` to your running of celery and flask.  For example to send all jobs to IP addresses `1.2.3.4` and `2.3.4.5`, do the following 2 commands in separate terminal tabs or windows:
+```
+DROPQ_WORKERS=1.2.3.4,2.3.4.5 BTAX_WORKERS=1.2.3.4,2.3.4.5 OGUSA_WORKERS=1.2.3.4,2.3.4.5 . run_celery_local.sh
+DROPQ_WORKERS=1.2.3.4,2.3.4.5 BTAX_WORKERS=1.2.3.4,2.3.4.5 OGUSA_WORKERS=1.2.3.4,2.3.4.5 . run_flask_server_local.sh
+```

deploy/README_puf_package.md

@@ -0,0 +1,63 @@
+# PUF package
+
+A private Anaconda package for Public Use File (PUF)
+
+## Preliminary
+
+### *No accidental uploads!*
+Make sure you never automatically upload this package to your personal
+account:
+
+`conda config --set anaconda_upload no`
+
+### Use Anaconda Token
+
+Get the *general* Anaconda upload token from me (Peter S) and save it in a one-line text file in your home directory with this file name:
+
+`.ospc_anaconda_token`
+
+Get the `conda:download` only Anaconda token from me (Peter S) and save it in a one-line text file in your home directory with this file name:
+
+`.ospc_anaconda_token_conda_download`
+
+
+## Get the PUF package builder
+
+I made a script that will automate creating a new package `puf` for a new `puf.csv` file.  To get the helper, download the private Anaconda pakcage `puf_builder`:
+
+```
+mkdir puf_work && cd puf_work
+anaconda --token ~/.ospc_anaconda_token download opensourcepolicycenter/puf_builder
+```
+The above downloads the builder using the token for privacy.  Then unzip the file you got:
+```
+unzip puf_builder.zip
+```
+Next run `build_puf_package.py` script with 3 arguments:
+ * Path to the new `puf.csv` you want in the new `puf` package
+ * A note about the new `puf.csv` (quotes around it if spaces in note)
+ * The new `puf` package's version, such as 0.0.0
+
+```
+python build_puf_package.py abc.txt "testing it out" 0.0.4
+```
+That will `conda build` the `puf` package after copying your new puf.csv into it, then upload packages to Anaconda under the private package:
+
+`opensourcepolicycenter/puf`
+
+## Make sure your new version was uploaded
+
+Check and make sure you have a `conda` type of package for the version you just created, for each of the major platforms:
+
+https://anaconda.org/opensourcepolicycenter/puf
+
+## Instaling the `puf` package
+
+Assuming you have saved the tokens mentioned at top of this page:
+
+```
+export TOKEN=`cat ~/.ospc_anaconda_token_conda_download`
+conda install -c https://conda.anaconda.org/t/$TOKEN/opensourcepolicycenter puf
+python -c "from puf import PUF, PUF_META, PUF_VERSION"
+
+```

deploy/ensure_procs_killed.sh

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+
+kill_from_args(){
+    if [ "$1" = "" ];then
+        echo Do nothing - give an arg like \"flask\" or \"celery\";
+        return 1;
+    fi
+    ps aux | grep "$1"  | grep 'python' | awk -F' ' '{print $2}' | xargs -n 1 kill -9;
+}
+kill_from_args $1
+
+

deploy/fab/copy_deploy_repo.py

@@ -0,0 +1,62 @@
+from __future__ import print_function
+import os
+import shutil
+import subprocess as sp
+import tempfile
+
+d = os.path.dirname
+DEPLOY_REPO = d(d(os.path.abspath(__file__)))
+REMOTE_DEPLOY = '/home/ubuntu/deploy'
+
+def check_unmodified():
+    args = 'git ls-files -m'.split()
+    proc = sp.Popen(args, stdout=sp.PIPE, stderr=sp.PIPE, cwd=DEPLOY_REPO, env=os.environ)
+    if proc.wait() != 0:
+        raise ValueError("Failed subproc {} - {} {}".format(args, proc.stdout.read(), proc.stderr.read()))
+    lines = proc.stdout.read().splitlines()
+    for line in lines:
+        if line.strip():
+            raise ValueError('There are modified files not checked in:\n {}'.format(lines))
+
+
+def copy_deploy_repo(sudo_func, put_func, run_func):
+    '''Copy the current deploy directory after ensuring all files
+    are committed and that only files within "git ls-files" are
+    copied over. Moves a zip and then unizps it to be /home/ubuntu/deploy
+    '''
+    tmp = os.path.join(tempfile.mkdtemp(), 'deploy')
+    try:
+        args = 'git ls-files'.split()
+        proc = sp.Popen(args, stdout=sp.PIPE, stderr=sp.PIPE, cwd=DEPLOY_REPO, env=os.environ)
+        if proc.wait() != 0:
+            raise ValueError("Failed subproc {} - {} {}".format(args, proc.stdout.read(), proc.stderr.read()))
+        lines = proc.stdout.read().splitlines()
+        for line in lines:
+            fname_rel = line.strip()
+            src = os.path.join(DEPLOY_REPO, fname_rel)
+            dst = os.path.join(tmp, fname_rel)
+            dirr = os.path.dirname(dst)
+            if not os.path.exists(dirr):
+                os.makedirs(dirr)
+            if os.path.exists(src):
+                shutil.copy(src, dst)
+            else:
+                print('WARNING:', src, '(deploy repo) is in git ls-files but not found.  Not copying...')
+        deploy_zip = os.path.join(os.path.dirname(tmp), 'deploy.zip')
+        args = 'zip -r {} ./*'.format(deploy_zip)
+        print(args, '(from {})'.format(tmp), os.listdir(tmp))
+        proc = sp.Popen(args, cwd=tmp, shell=True)
+        if proc.wait() or not os.path.exists(deploy_zip):
+            raise ValueError('Failed on zipping in {}'.format(tmp))
+        print('COPY DEPLOY REPO')
+        if sudo_func:
+            sudo_func('apt-get install -y unzip')
+        run_func('rm -rf {}'.format(REMOTE_DEPLOY))
+        run_func('mkdir {}'.format(REMOTE_DEPLOY))
+        put_func(deploy_zip, os.path.join(REMOTE_DEPLOY, 'deploy.zip'))
+        run_func('cd {} && unzip deploy.zip'.format(REMOTE_DEPLOY)) # TODO && rm deploy.zip?
+        remote_reset_server = os.path.join(os.path.dirname(REMOTE_DEPLOY), 'reset_server.sh')
+        run_func('cd {} && cp reset_server.sh {}'.format(os.path.join(REMOTE_DEPLOY, 'fab'), remote_reset_server))
+    finally:
+        shutil.rmtree(os.path.dirname(tmp))
+

deploy/fab/dropq.conf

@@ -0,0 +1,13 @@
+#
+# A virtual host using mix of IP-, name-, and port-based configuration
+#
+
+server {
+    listen 80;
+    server_name dropq.ospc.org;
+    location / {
+        proxy_set_header   X-Real-IP $remote_addr;
+        proxy_set_header   Host      $http_host;
+        proxy_pass         http://127.0.0.1:5050;
+    }
+}

deploy/fab/dropq_environment.yml

@@ -0,0 +1,19 @@
+name: aei_dropq
+dependencies:
+- python=2.7.9
+- pandas>=0.18
+- flask
+- greenlet
+- bokeh
+- xlrd
+- pip:
+  - fabric
+  - boto
+  - celery
+  - eventlet
+  - retrying
+  - plotly
+channels:
+  - ospc
+  - amfarrell
+  - defaults

deploy/fab/ec2-deploy.sh

@@ -0,0 +1 @@
+USE_LATEST_KEY='True' EC2_INSTANCE_TYPE='m3.large' SERVER_FILE_PATH="XXX" PUF_FILE_PATH="XXX" GITHUB_USERNAME='ospc-auto' GITHUB_PRIVATE_KEY_PATH='XXX/ospc-auto/id_rsa' OGUSA_GIT_BRANCH='XXX' DROPQ_GIT_BRANCH='XXX' DEPLOY_GIT_BRANCH='XXX' AWS_ID='XXX' AWS_SECRET='XXX' python ec2-fabfile.py