Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding docker tools and instructions for client side devs #854

Merged
merged 2 commits into from
Dec 4, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions client/docker/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Install 311 Server locally for development

This doc in intended for client side developers working on the 311 Data project who want an easy way to run a local copy of the server. It downloads images for whatever code is in the ```dev``` branch in GitHub.

## Introduction and Prerequisites

These instructions assume only that Docker is installed. They also assume the steps are being followed on a Mac, but it should be easy enough to follow for other platforms.

All of the 311 Server components (e.g. Postgres, Redis) are downloaded and run as Docker containers. This wouldn't be the best configuration for a production system, but it is more than adequate for development.

The 4 included images will need about 1 GB of space with an additional 1 GB or so needed for the data.

## Initialization Script
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about swapping the positions of Initialization Script and Installation Instructions sections? Devs (me) who don't read through the whole README first will see instructions first then an explanation of what the init script does.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was confused if i had to create the Initialization script myself. Would be good to just have the install instructions to make it simpler but otherwise everything works when executing the bash script. Awesome work!


```bash
cp api.env .env # copy the sample variables to .env
docker-compose up -d # downloads and starts all images
docker-compose run api alembic upgrade head # install the database schema
docker-compose run prefect python flow.py # loads data to database
open http://localhost:5000/docs # open the docs page (on a Mac)
```

## Installation Instructions

The easiest way to install the 311 Server is with the included initialization script. Open a terminal to the client/docker directory and execute the following commands.

```bash
chmod +x init_api.sh # make the init bash script executable
./init_api.sh # execute the script
```

The process should take about 5 minutes to run from start to finish.

## Updating the data

In order to have current data for reports the prefect container should be run periodically. It will get any data added or changed since the last time it was run.

```bash
docker-compose run prefect python flow.py # loads new data to database
```
28 changes: 28 additions & 0 deletions client/docker/api.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@

######################### DOCKER-COMPOSE SETTINGS ########################

COMPOSE_PROJECT_NAME=311_api

# host/public configs
API_HOST_PORT=5000
DB_HOST_PORT=5432

# api config
DEBUG=True
API_RESTART_POLICY=no
APP_PROTOCOL=http
APP_HOST=api
APP_PORT=5000

# db connection config
DB_HOST=db
DB_PORT=5432
DB_USER=311_user
DB_PASS=311_pass
DB_NAME=311_db
DB_ECHO=False

############################ PREFECT CONFIG ##############################

# to populate a new database for development
PREFECT__DATA__YEARS=2020,2019
52 changes: 52 additions & 0 deletions client/docker/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
version: '3.8'

services:
db:
container_name: 311-postgres
image: postgres:12
restart: always
env_file: .env
environment:
POSTGRES_USER: ${DB_USER}
POSTGRES_PASSWORD: ${DB_PASS}
POSTGRES_DB: ${DB_NAME}
ports:
- target: 5432
published: ${DB_HOST_PORT}
volumes:
- backend_data:/var/lib/postgresql/data

redis:
container_name: 311-redis
image: redis:6
restart: always
ports:
- "6379:6379"

api:
container_name: 311-api
image: la311data/311_data_api:dev
restart: ${API_RESTART_POLICY}
env_file: .env
environment:
DATABASE_URL: postgresql://${DB_USER}:${DB_PASS}@db:5432/${DB_NAME}
CACHE_ENDPOINT: redis
ports:
- target: ${APP_PORT}
published: ${API_HOST_PORT}
depends_on:
- db
- redis

prefect:
container_name: 311-prefect
image: la311data/311_data_prefect:dev
env_file: .env
environment:
PREFECT__CONTEXT__SECRETS__DSN: postgresql://${DB_USER}:${DB_PASS}@${DB_HOST}:5432/${DB_NAME}
PREFECT__API_URL: ${APP_PROTOCOL}://${APP_HOST}:${APP_PORT}
depends_on:
- db

volumes:
backend_data:
6 changes: 6 additions & 0 deletions client/docker/init_api.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/bin/bash
cp api.env .env
docker-compose up -d # downloads and starts all images (in background)
docker-compose run api alembic upgrade head # install the database schema
docker-compose run prefect python flow.py # loads data to database
open http://localhost:5000/docs # open the docs page (on a Mac)