Skip to content

Commit

Permalink
Merge pull request #1072 from hackforla/mattyweb/issue1071
Browse files Browse the repository at this point in the history
Remove Postman tests
  • Loading branch information
mattyweb authored Apr 6, 2021
2 parents 733b2a9 + 617932b commit 8b65a87
Show file tree
Hide file tree
Showing 9 changed files with 26 additions and 1,435 deletions.
39 changes: 26 additions & 13 deletions server/README.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,30 @@
# 311 Data Server

[![Build and Test Action Status](https://github.com/hackforla/311-data/workflows/Build%20and%20Test/badge.svg)](https://github.com/hackforla/311-data/actions)
[![API Build and Test Action Status](https://github.com/hackforla/311-data/workflows/Build%20and%20Test/badge.svg)](https://github.com/hackforla/311-data/actions)

## Server Tech Stack

The 311 Data Server is comprised of several different components deployed in containers to the cloud using Terraform.
The 311 Data Server is comprised of several different components deployed in containers to the public cloud. The API is intended to be modern, asynchronous, simple, performant, secure, open-source and well-supported.

- [FastAPI](https://fastapi.tiangolo.com/) and [Starlette](https://www.starlette.io/): tools to provide a fast, asynchronous stack for building RESTful APIs in Python
- [Gino](https://python-gino.org/) and [SQL Alchemy](https://www.sqlalchemy.org/): tools for accessing and modifying the database
- [Postgres](https://www.postgresql.org/docs/12/index.html): the persistent SQL database
- [Redis](https://redis.io/) with [Aiocache](https://aiocache.readthedocs.io/) and [Aioredis](https://aioredis.readthedocs.io/en/v1.3.0/): to provide high-performance caching
- [Prefect](https://www.prefect.io/core): for the nightly data ingestion pipeline
The server code is set up as a monorepo with directories and containers corresponding to the following functions:

Everything runs on Python 3.7 and the Debian 10 "Buster" Linux distribution where applicable.
- RESTful Python API (/api)
- [FastAPI](https://fastapi.tiangolo.com/) and [Starlette](https://www.starlette.io/): tools to provide a fast, asynchronous stack for building RESTful APIs in Python
- [Gino](https://python-gino.org/) and [SQL Alchemy](https://www.sqlalchemy.org/): tools for accessing and modifying the database
- [Postgres](https://www.postgresql.org/docs/12/index.html): the persistent SQL database
- [Redis](https://redis.io/) with [Aiocache](https://aiocache.readthedocs.io/) and [Aioredis](https://aioredis.readthedocs.io/en/v1.3.0/): to provide high-performance caching
- Report Server (/dash)
- [Plotly Dash](https://dash.plotly.com/): for interactive, responsive reports
- [Pandas](https://pandas.pydata.org/): for pulling, aggregating, and calculating data in reports
- Data loading pipeline (/prefect)
- [Prefect](https://www.prefect.io/core): for the nightly data ingestion tasks and cache clearing and priming

The last 2 projects are:

- [Terraform](https://www.terraform.io/): for managing the AWS infrastructure
- [Locust](https://locust.io/): for load-testing the API

The API is intended to be modern, simple, performant, secure, open-source and well-supported. The tech stack focuses on Python tools with good asynchronous support. As asynchronous libraries are still somewhat immature these dependencies may need to be updated in the future.
Everything runs on Python 3.7 and the Debian 10 "Buster" Linux distribution where applicable.

## Data Loading

Expand All @@ -24,21 +34,24 @@ For more information about the data loading process look at the [Prefect README]

## Testing

Testing is done locally and as a CI process run using GitHub Actions.
Unit and integration testing is done locally and as a CI process run using GitHub Actions.

There are several testing tools used in the project:

- pytest: unit and integration tests
- postman: REST API integration tests
- locust: load tests

We currently have 90+% coverage on the API. To get code coverage reports run ```pytest --cov=code```

## Deployment

The API is deployed to AWS using Terraform. API containers are hosted in ECS with data served from an RDS instance.
### Code Deployment

Containers are deployed on merge to Docker Hub and new tasks are created in ECS. The result is a zero-downtime deployment. When a new version requires a database change a separate task must be created to run upgrades using Alembic.

### Infrastructure Deployment

For more information about the deployment process look at the [Terraform README](terraform/README.md).
The infrastructure is deployed to AWS using Terraform. API containers are hosted in ECS with data served from an RDS instance. For more information about the infrastrucure deployment process look at the [Terraform README](terraform/README.md).

## More Information

Expand Down
52 changes: 0 additions & 52 deletions server/postman/README.md

This file was deleted.

Loading

0 comments on commit 8b65a87

Please sign in to comment.