Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

changes to enable module use #787

Merged
merged 3 commits into from
Aug 23, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions server/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
############################### API SETTINGS #############################

# Server
HOST=0.0.0.0
PORT=5000
DEBUG=0
ACCESS_LOG=1
Expand Down
3 changes: 2 additions & 1 deletion server/.gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
.env
*.env

/api/__tmp__
/jmeter/out

__pycache__
.pytest_cache
61 changes: 40 additions & 21 deletions server/README.md
Original file line number Diff line number Diff line change
@@ -1,71 +1,87 @@
## Getting Started
# Getting Started

### TL;DR
- install docker and docker compose on your machine
- run `chmod +x install.sh && ./install.sh` from this directory
- put a [Socrata](https://dev.socrata.com/) api token in your `.env` file
- run `docker-compose run api python bin/db_seed.py --years 2019,2020`
## TL;DR

- install docker and docker compose on your machine
- run `chmod +x install.sh && ./install.sh` from this directory
- put a [Socrata](https://dev.socrata.com/) API token in your `.env` file
- run `docker-compose run api python bin/db_seed.py --years 2019,2020`

### Step 1: install docker and docker-compose

Docker and Docker-Compose are the only required dependencies for running the server. You can find installation instructions [here](https://docs.docker.com/compose/install/) for Docker and [here](https://docs.docker.com/compose/install/) for Docker-Compose.

Once you're done, you can check that everything is working:
```

```bash
docker --version # confirms docker is installed
docker-compose --version # confirms docker-compose is installed
docker info # confirms the docker daemon is running
```

### Step 2: run the install script

With docker running on your machine, `cd` into this directory and run:
```

```bash
chmod +x install.sh && ./install.sh
```
This will download/build a bunch of docker images, create your `.env` file, set up your database, and then fire everything up. If all goes well, at the end you should have running api server backed by Postgres and Redis. You can then tour the running services by hitting these URLs in a browser:
- http://localhost:5000 -- the api -- should say "you hit the index". This means the api is running.
- http://localhost:8080 -- a postgres GUI. Login with the following:

This will download/build a bunch of docker images, create your `.env` file, set up your database, and then fire everything up. If all goes well, at the end you should have running API server backed by Postgres and Redis. You can then tour the running services by hitting these URLs in a browser:

- http://localhost:5000 -- the API -- should say "you hit the index". This means the API is running.
- http://localhost:8080 -- a Postgres GUI. Login with the following:
- System: **PostgreSQL**
- Server: **db**
- Username: **311_user**
- Password: **311_pass**
- Database: **311_db**
- http://localhost:5001 -- a redis GUI. Login with the following:
- http://localhost:5001 -- a Redis GUI. Login with the following:
- Host: **redis**
- Port: **6379**
- Database ID: **0**

You can shut down all the services by hitting `Ctrl-C`. And run `docker-compose up` to bring everything back up again.

### Step 3: seed your database

Right now the server is functional and all endpoints should be working. But for most purposes you'll need some data in your database. The data comes from [Socrata](https://dev.socrata.com/), a public api that hosts many datasets for LA and other cities. To add data, you'll need to get a Socrata token and run one more command.

- #### 3a: add a Socrata token to your .env file (possible optional)

Socrata threatens to throttle you if you don't have an api token. We're not sure they actually do that, but the api does seem to run more slowly without a token. So get a token from another team member, or [register](https://opendata.socrata.com/login) with Socrata and they'll give you one. Then add it to the Socrata section in your `.env` file:
```

```bash
SOCRATA_TOKEN=thetoken
```

- #### 3b: run the seed script

It takes a while to seed the database, so we recommend starting with 2 years of data from Socrata. Run the following command to get data for 2019 and 2020, which is plenty for most dev purposes. ETA **20 to 30 minutes**. (If you've still got the backend services running, you can run this command in a separate terminal window.)
```

```bash
docker-compose run api python bin/db_seed.py --years 2019,2020
```

If you decide later that you need more data, just run the command again with the year(s) you want to add. Socrata goes back to 2015.

(NOTE: run the above command with `--help` instead of `--years` for more info on the options. And if you ever want to reset your database and start over, run `docker-compose run api python bin/db_reset.py`.)


## Development

### Optional Dependencies

- #### Postman
#### Postman

[Postman](https://www.postman.com/) is an api-development tool that lets you save api requests so you can run them over and over without having to remember what all the parameters are (or use the awkward syntax of `curl`). You can also group api calls together in collections, and run all of the api calls in a collection at once -- which is a great way to test the entire api. If you'd like to use it, see the README in `/server/postman`, which explains how to set it up with a collection that contains pre-defined api calls for all of our endpoints.

- #### Jmeter
[Jmeter](https://jmeter.apache.org/) is a performance-focused api testing tool. It produces reports that show how the api performs when multiple users are using the app simultaneously. For more info, see the README in `/server/jmeter`.
#### JMeter

[JMeter](https://jmeter.apache.org/) is a performance-focused api testing tool. It produces reports that show how the api performs when multiple users are using the app simultaneously. For more info, see the README in `/server/jmeter`.

### Useful commands
```

```bash
docker-compose up # start the backend services
docker-compose up --build # start the backend services after rebuilding containers

Expand All @@ -75,7 +91,8 @@ docker-compose run api pytest # run unit tests against python code
```

### Using the python interpreter
```

```bash
docker-compose run api bash
cd src
python
Expand All @@ -93,7 +110,9 @@ Type "help", "copyright", "credits" or "license" for more information.
```

## Uninstall

Run this command to remove all docker containers, images, volumes, and networks that are specific to this project. It will leave in place generic docker assets (like the `postgres` and `python` images) that you may be using for other purposes.
```

```bash
docker-compose down --rmi local --volumes
```
61 changes: 61 additions & 0 deletions server/api/Pipfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
aiofiles = "==0.4.0"
attrs = "==19.3.0"
certifi = "==2019.11.28"
chardet = "==3.0.4"
entrypoints = "==0.3"
flake8 = "==3.7.9"
h11 = "==0.8.1"
h2 = "==3.1.1"
hpack = "==3.0.0"
httpcore = "==0.3.0"
httptools = "==0.0.13"
hyperframe = "==5.2.0"
idna = "==2.9"
importlib-metadata = "==1.4.0"
mccabe = "==0.6.1"
more-itertools = "==8.1.0"
multidict = "==4.5.2"
numpy = "==1.18.1"
pandas = "==1.0.1"
pluggy = "==0.13.1"
psycopg2-binary = "==2.8.4"
py = "==1.8.1"
pycodestyle = "==2.5.0"
pyflakes = "==2.1.1"
pyparsing = "==2.4.6"
pysupercluster = "==0.7.6"
pytest = "==5.3.3"
python-dateutil = "==2.8.1"
python-dotenv = "==0.13.0"
python-http-client = "==3.2.7"
pytz = "==2019.3"
redis = "==3.5.0"
requests = "==2.23.0"
requests-async = "==0.5.0"
rfc3986 = "==1.3.2"
sanic = "==19.9.0"
sendgrid = "==6.3.1"
six = "==1.14.0"
sodapy = "==2.0.0"
tabulate = "==0.8.7"
ujson = "==1.35"
urllib3 = "==1.25.8"
uvloop = "==0.14.0"
wcwidth = "==0.1.8"
websockets = "==8.1"
zipp = "==1.0.0"
sanic_compress = "==0.1.1"
Sanic-Cors = "==0.10.0.post3"
Sanic-Plugins-Framework = "==0.9.2"
SQLAlchemy = "==1.3.13"

[requires]
python_version = "3.7"
Loading