Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add prefect #823

Merged
merged 96 commits into from
Nov 2, 2020
Merged
Changes from 1 commit
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
8f7b6ab
First set of API refactor changes
Aug 27, 2020
d4c65fa
added pins route and Gzip
Aug 28, 2020
63cc3b7
first draft of legacy endpoints
Sep 1, 2020
8886ca7
fixing lint issues from flake8
Sep 1, 2020
35628ff
added support for deploying new API in Docker
Sep 2, 2020
bd4ef1e
new date parsing and config changes
Sep 2, 2020
cc995e0
more docker and config changes plus README
Sep 3, 2020
58a340a
default LEGACY_MODE to True
Sep 3, 2020
d1b5daf
DB seed needs database URL
Sep 3, 2020
e53dbab
YAML fix
Sep 3, 2020
f32f628
compose networking
Sep 3, 2020
11b9441
set test default config
Sep 3, 2020
aef7f67
updated missing .env.example settings
Sep 3, 2020
a4cb368
postman tests accept either 400 or 422 for bad input
jmensch1 Sep 3, 2020
5b916f8
comparison set validation
jmensch1 Sep 4, 2020
a1f94c7
model models and added simple entity cache
Sep 8, 2020
986ba3e
added cache status check
Sep 8, 2020
6ed9777
fixing flake8
Sep 8, 2020
1dcf5f5
disabling cache build
Sep 8, 2020
3911501
first version of alembic
Sep 9, 2020
d7bc8f3
added testing
Sep 11, 2020
e4b4fc7
updating ENV and requirements
Sep 11, 2020
69fec6c
turned testing off
Sep 11, 2020
5316119
test tweaks
Sep 11, 2020
d8379dd
dockerignore changes
Sep 11, 2020
fa68515
adding cwd to pythonpath
Sep 11, 2020
05ac6ab
fixed db env settings
Sep 11, 2020
0c473fa
setting db to test in env
Sep 11, 2020
c6e7a79
don't add _test to db name if there
Sep 11, 2020
b8795b9
more reliable tests
Sep 11, 2020
6126a68
more test cleanup
Sep 11, 2020
e9db6da
always add map/vis views
Sep 12, 2020
52baa31
replacing db tests
Sep 12, 2020
370de62
setting testing
Sep 12, 2020
8fe0c22
more test cleanup
Sep 12, 2020
6e62c67
fix flake8
Sep 12, 2020
40dac4e
updated libraries
Sep 12, 2020
3d2d5f7
slimming docker image
Sep 12, 2020
8170c3c
removed lru decorator
Sep 12, 2020
e44b669
logging response text on postman test failure
jmensch1 Sep 12, 2020
d603132
Merge branch 'add-alembic' of https://github.com/hackforla/311-data i…
jmensch1 Sep 12, 2020
193b5ca
postman logging
jmensch1 Sep 13, 2020
155350f
running CI on push
jmensch1 Sep 13, 2020
a9c970a
more postman logging
jmensch1 Sep 13, 2020
a66ea53
commented build_cache
jmensch1 Sep 13, 2020
a066a0c
linting
jmensch1 Sep 13, 2020
1f0df0f
logging 500 errors for all positive postman tests
jmensch1 Sep 13, 2020
f40a7cd
revert to 2 databases
Sep 16, 2020
60702ed
Merge branch 'add-alembic' of https://github.com/hackforla/311-data i…
Sep 16, 2020
7c05209
testing to false
Sep 16, 2020
2205bc7
remove views from db_seed
Sep 16, 2020
d846040
removed db reset
Sep 16, 2020
9c09fd5
added data file
Sep 16, 2020
070d75e
removing TESTING env
Sep 16, 2020
65a5335
removing from api.env
Sep 16, 2020
fa7de52
fixing flake8
Sep 16, 2020
8dc6a0b
error improvement
Sep 16, 2020
711b39f
1 more change
Sep 16, 2020
60589b0
undo docker compose change
Sep 16, 2020
d649c45
test database logging
Sep 18, 2020
a89155b
keep test db and new db fixture
Sep 20, 2020
a2698cf
seed file and better alembic logging
Sep 20, 2020
fd0e99c
adding env for seed
Sep 20, 2020
af2135d
fix env
Sep 20, 2020
d53f499
weird cluster fix
Sep 20, 2020
a59fe2f
LAST_WEEK to YEAR_TO_DATE in cypress
Sep 20, 2020
87ffbd0
type selector change
Sep 20, 2020
ebd5b1b
direct request type all
Sep 20, 2020
6829636
handling 10K+
Sep 20, 2020
46cd64a
adding coverage
Sep 20, 2020
8fc30d1
added async redis caching
Sep 22, 2020
2eabea3
flake fix
Sep 22, 2020
8dd1c75
turning off legacy mode
Sep 22, 2020
ac21516
added map/pins placeholder
Sep 22, 2020
0a8a262
added cache retries, timeout + pool changes
Sep 23, 2020
39c2c9f
api status check and test
Sep 23, 2020
9c1caab
adding sentry
Sep 23, 2020
559366a
sentry tweaks
Sep 23, 2020
efce220
remove weirdness
Sep 23, 2020
e846e9e
fixing requirements
Sep 23, 2020
c29a8ba
works with new UI, pins, open-requests
Sep 25, 2020
edc8616
fixed date format problem
Sep 25, 2020
1ecdfd3
using gunicorn for API
Sep 25, 2020
7f60721
added caching to open-request and reset-cache tool
Sep 29, 2020
3a26baa
porting github and email services
Oct 6, 2020
b6a2aed
refactor status service
Oct 6, 2020
1143401
utilities refactor
Oct 6, 2020
d49b8b9
ported reports
Oct 6, 2020
41a4763
fixed bug in time reports
Oct 7, 2020
8479dfc
removed legacy service calls
Oct 7, 2020
05ea638
made SENTRY URL configurable
mattyweb Oct 19, 2020
decee73
adding SENTRY URL
mattyweb Oct 19, 2020
229e9b4
added feedback test w/201 code
mattyweb Oct 19, 2020
bcf930b
minor refactoring and new routes
mattyweb Oct 19, 2020
0027fa7
added TODOs
mattyweb Oct 19, 2020
d1a27b2
adding prefect data loading
mattyweb Oct 23, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
First set of API refactor changes
Matthew Webster committed Aug 27, 2020
commit 8f7b6abbdeadf0bb9dc1bc8a0f08076e6c5d2e11
4 changes: 4 additions & 0 deletions server/api/Pipfile
Original file line number Diff line number Diff line change
@@ -24,6 +24,10 @@ sendgrid = "*"
sodapy = "*"
SQLAlchemy = "*"
tabulate = "*"
gino = {extras = ["starlette"], version = "*"}
fastapi = "*"
uvicorn = "*"
gunicorn = "*"

[requires]
python_version = "3.7"
151 changes: 146 additions & 5 deletions server/api/Pipfile.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Empty file.
3 changes: 3 additions & 0 deletions server/api/code/311_data_api/asgi.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from .main import get_app

app = get_app()
33 changes: 33 additions & 0 deletions server/api/code/311_data_api/config.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
from sqlalchemy.engine.url import URL, make_url
from starlette.config import Config
from starlette.datastructures import Secret

config = Config(".env")

DB_DRIVER = config("DB_DRIVER", default="postgresql")
DB_HOST = config("DB_HOST", default=None)
DB_PORT = config("DB_PORT", cast=int, default=None)
DB_USER = config("DB_USER", default=None)
DB_PASSWORD = config("DB_PASSWORD", cast=Secret, default=None)
DB_DATABASE = config("DB_DATABASE", default=None)
DB_DSN = config(
"DB_DSN",
cast=make_url,
default=URL(
drivername=DB_DRIVER,
username=DB_USER,
password=DB_PASSWORD,
host=DB_HOST,
port=DB_PORT,
database=DB_DATABASE,
),
)
DB_POOL_MIN_SIZE = config("DB_POOL_MIN_SIZE", cast=int, default=1)
DB_POOL_MAX_SIZE = config("DB_POOL_MAX_SIZE", cast=int, default=16)
DB_ECHO = config("DB_ECHO", cast=bool, default=False)
DB_SSL = config("DB_SSL", default=None)
DB_USE_CONNECTION_FOR_REQUEST = config(
"DB_USE_CONNECTION_FOR_REQUEST", cast=bool, default=True
)
DB_RETRY_LIMIT = config("DB_RETRY_LIMIT", cast=int, default=32)
DB_RETRY_INTERVAL = config("DB_RETRY_INTERVAL", cast=int, default=1)
36 changes: 36 additions & 0 deletions server/api/code/311_data_api/main.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
import logging

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware

from .routers import index, councils, regions, request_types, service_requests
from .models import db

try:
from importlib.metadata import entry_points
except ImportError: # pragma: no cover
from importlib_metadata import entry_points

logger = logging.getLogger(__name__)


def get_app():
app = FastAPI(title="311 Data API")

db.init_app(app)

app.include_router(index.router)
app.include_router(councils.router, prefix="/councils")
app.include_router(regions.router, prefix="/regions")
app.include_router(request_types.router, prefix="/types")
app.include_router(service_requests.router, prefix="/requests")

app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)

return app
14 changes: 14 additions & 0 deletions server/api/code/311_data_api/models/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
from gino.ext.starlette import Gino

from .. import config

db = Gino(
dsn=config.DB_DSN,
pool_min_size=config.DB_POOL_MIN_SIZE,
pool_max_size=config.DB_POOL_MAX_SIZE,
echo=config.DB_ECHO,
ssl=config.DB_SSL,
use_connection_for_request=config.DB_USE_CONNECTION_FOR_REQUEST,
retry_limit=config.DB_RETRY_LIMIT,
retry_interval=config.DB_RETRY_INTERVAL,
)
9 changes: 9 additions & 0 deletions server/api/code/311_data_api/models/council.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
from . import db


class Council(db.Model):
__tablename__ = "councils"

council_id = db.Column(db.SmallInteger, primary_key=True)
council_name = db.Column(db.String)
region_id = db.Column(db.SmallInteger)
8 changes: 8 additions & 0 deletions server/api/code/311_data_api/models/region.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
from . import db


class Region(db.Model):
__tablename__ = 'regions'

region_id = db.Column(db.SmallInteger, primary_key=True)
region_name = db.Column(db.String)
8 changes: 8 additions & 0 deletions server/api/code/311_data_api/models/request_type.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
from . import db


class RequestType(db.Model):
__tablename__ = 'request_types'

type_id = db.Column(db.SmallInteger, primary_key=True)
type_name = db.Column(db.String)
14 changes: 14 additions & 0 deletions server/api/code/311_data_api/models/service_request.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
from . import db


class ServiceRequest(db.Model):
__tablename__ = 'service_requests'

request_id = db.Column(db.Integer, primary_key=True)
created_date = db.Column(db.Date)
closed_date = db.Column(db.Date)
type_id = db.Column(db.SmallInteger)
council_id = db.Column(db.SmallInteger)
address = db.Column(db.String)
latitude = db.Column(db.Float)
longitude = db.Column(db.Float)
Empty file.
31 changes: 31 additions & 0 deletions server/api/code/311_data_api/routers/councils.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
from typing import List

from fastapi import APIRouter
from pydantic import BaseModel

from ..models.council import Council
from ..models import db

router = APIRouter()


class CouncilModel(BaseModel):
council_id: int
council_name: str

class Config:
orm_mode = True


Items = List[CouncilModel]


@router.get("/", response_model=Items)
async def index():
return await db.all(Council.query)


@router.get("/{cid}")
async def get_council(cid: int):
council = await Council.get_or_404(cid)
return council.to_dict()
95 changes: 95 additions & 0 deletions server/api/code/311_data_api/routers/index.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
from typing import List, Optional
import datetime

from fastapi import responses
from fastapi import APIRouter
from pydantic import BaseModel

from services import status, map, visualizations, requests

router = APIRouter()


class Bounds(BaseModel):
north: float
south: float
east: float
west: float


class Filter(BaseModel):
startDate: str
endDate: str
ncList: List[int]
requestTypes: List[str]
zoom: Optional[int] = None
bounds: Optional[Bounds] = None


@router.get("/")
async def index():
return {"message": "Hello, new index!"}


@router.get("/status/api")
async def status_api():
result = await status.api()
return result


@router.get("/status/db")
async def status_db():
result = await status.database()
return result


@router.get("/status/system")
async def status_system():
result = await status.system()
return result


@router.get("/servicerequest/{srnumber}")
async def get_service_request_by_string(srnumber: str):
# result = await get_service_request(int(srnumber[2:]))
result = requests.item_query(srnumber)
return result


@router.post("/map/clusters")
async def get_clusters(filter: Filter):
start_time = datetime.datetime.strptime(filter.startDate, '%m/%d/%Y')
end_time = datetime.datetime.strptime(filter.endDate, '%m/%d/%Y')

result = await map.pin_clusters(start_time,
end_time,
filter.requestTypes,
filter.ncList,
filter.zoom,
dict(filter.bounds)
)
return result


@router.post("/map/heat")
async def get_heatmap(filter: Filter):
start_time = datetime.datetime.strptime(filter.startDate, '%m/%d/%Y')
end_time = datetime.datetime.strptime(filter.endDate, '%m/%d/%Y')

result = await map.heatmap(startDate=start_time,
endDate=end_time,
requestTypes=filter.requestTypes,
ncList=filter.ncList)
return responses.JSONResponse(result.tolist())


@router.post("/visualizations")
async def get_visualizations(filter: Filter):
start_time = datetime.datetime.strptime(filter.startDate, '%m/%d/%Y')
end_time = datetime.datetime.strptime(filter.endDate, '%m/%d/%Y')

result = await visualizations.visualizations(startDate=start_time,
endDate=end_time,
requestTypes=filter.requestTypes,
ncList=filter.ncList)
return result
31 changes: 31 additions & 0 deletions server/api/code/311_data_api/routers/regions.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
from typing import List

from fastapi import APIRouter
from pydantic import BaseModel

from ..models.region import Region
from ..models import db

router = APIRouter()


class RegionModel(BaseModel):
region_id: int
region_name: str

class Config:
orm_mode = True


Items = List[RegionModel]


@router.get("/", response_model=Items)
async def index():
return await db.all(Region.query)


@router.get("/{rid}")
async def get_region(rid: int):
region = await Region.get_or_404(rid)
return region.to_dict()
31 changes: 31 additions & 0 deletions server/api/code/311_data_api/routers/request_types.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
from typing import List

from fastapi import APIRouter
from pydantic import BaseModel

from ..models.request_type import RequestType
from ..models import db

router = APIRouter()


class RequestTypeModel(BaseModel):
type_id: int
type_name: str

class Config:
orm_mode = True


Items = List[RequestTypeModel]


@router.get("/", response_model=Items)
async def index():
return await db.all(RequestType.query)


@router.get("/{tid}")
async def get_request_type(tid: int):
request_type = await RequestType.get_or_404(tid)
return request_type.to_dict()
44 changes: 44 additions & 0 deletions server/api/code/311_data_api/routers/service_requests.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
from typing import List
import datetime

from fastapi import APIRouter
from pydantic import BaseModel

from ..models.service_request import ServiceRequest
from ..models import db

router = APIRouter()


class ServiceRequestModel(BaseModel):
request_id: int
council_id: int
type_id: int
created_date: datetime.date
closed_date: datetime.date
address: str
latitude: float
longitude: float

class Config:
orm_mode = True


Items = List[ServiceRequestModel]


@router.get("/", response_model=Items)
async def index(skip: int = 0, limit: int = 100):
async with db.transaction():
cursor = await ServiceRequest.query.gino.iterate()
if skip > 0:
await cursor.forward(skip) # skip 80 rows
result = await cursor.many(limit) # and retrieve next 10 rows

return result


@router.get("/{srid}")
async def get_service_request(srid: int):
svcreq = await ServiceRequest.get_or_404(srid)
return svcreq.to_dict()
13 changes: 13 additions & 0 deletions server/api/code/run.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import os

if __name__ == "__main__":
import uvicorn
import sys

print(sys.path)

uvicorn.run(
"311_data_api.asgi:app",
host=os.getenv("APP_HOST", "127.0.0.1"),
port=int(os.getenv("APP_PORT", "5000")),
)
73 changes: 72 additions & 1 deletion server/api/src/db/requests/views.py
Original file line number Diff line number Diff line change
@@ -66,12 +66,83 @@ def create():
""")

# TODO: this is a potential new refactor of the database not currently in API
exec_sql("""
CREATE MATERIALIZED VIEW regions AS
WITH join_table AS (
select distinct apc
from requests
where apc is not null
) SELECT
ROW_NUMBER () OVER (order by apc) as region_id,
left(apc, -4) as region_name
FROM join_table;
CREATE UNIQUE INDEX ON regions(region_id);
------------------------------------------
CREATE MATERIALIZED VIEW councils AS
SELECT DISTINCT on (nc)
nc as council_id,
ncname as council_name,
region_id
FROM requests
LEFT JOIN regions ON left(requests.apc, -4) = regions.region_name
WHERE requests.nc is not null
ORDER BY nc, createddate desc;
CREATE UNIQUE INDEX ON councils(council_id);
CREATE INDEX ON councils(region_id);
--add shortname, geometry, centroid, d and w websites
------------------------------------------
CREATE MATERIALIZED VIEW request_types AS
WITH join_table AS (
select distinct requesttype
from requests
) SELECT
ROW_NUMBER () OVER (order by requesttype) as type_id,
requesttype as type_name
FROM join_table;
CREATE UNIQUE INDEX ON request_types(type_id);
--colors: primary/alt, abbreviations
------------------------------------------
CREATE MATERIALIZED VIEW service_requests AS
SELECT right(requests.srnumber::VARCHAR(12), -2)::INTEGER as request_id,
requests.createddate::DATE as created_date,
requests.closeddate::DATE as closed_date,
request_types.type_id as type_id,
requests.nc::SMALLINT as council_id,
requests.address::VARCHAR(100),
requests.latitude,
requests.longitude
FROM requests, request_types
WHERE
requests.latitude IS NOT NULL
AND requests.longitude IS NOT NULL
AND type_name = requests.requesttype;
CREATE UNIQUE INDEX ON service_requests(request_id);
CREATE INDEX ON service_requests(created_date);
CREATE INDEX ON service_requests(type_id);
CREATE INDEX ON service_requests(council_id);
""")


def refresh():
log('\nRefreshing views')
exec_sql("""
REFRESH MATERIALIZED VIEW CONCURRENTLY map;
REFRESH MATERIALIZED VIEW CONCURRENTLY vis;
REFRESH MATERIALIZED VIEW CONCURRENTLY open_requests;
REFRESH MATERIALIZED VIEW CONCURRENTLY service_requests;
""")