Docs cleanup 2021

README.md

@@ -1,72 +1,59 @@
-# 311-data
-Empower Neighborhood Associations to Improve analysis of their initiatives using 311 data
+# The 311 Data project
 
-## Frontend Technologies
-  * Javascript (React)
-  * Redux
-  * Bulma
-  * HTML/CSS
+Empowering Los Angeles neighborhood councils to analyze initiatives using data collected from the city 311 system.
 
-## Backend Technologies
+![311 Screenshot](docs/images/screenshot.png)
 
-  * FastAPI: asynchronous Python REST API
-  * Redis: key-value caching
-  * PostgreSql: persistant SQL database
-  * Prefect: data ingestion pipeline in Python
-  * Docker: containerized servers hosted in AWS
+## Project Technology
 
-## Data Analysis Technologies
-  * Python
-  * Pandas
-  * Numpy
-  * PostgreSql
-  * Socrata API
+### Frontend
 
-## UI/UX Technologies
-  * Figma
-  * Google Drive
-  * Adobe CC
-  * Miro
+* Javascript (React)
+* Redux
+* Bulma
+* HTML/CSS
 
-## Management Philosophies:
-  * Open Source
-  * Kanban
-
-## To Sign Up:
-  * Follow the steps [here](https://github.com/hackforla/311-data/wiki/Joining-the-311-Team)  
-  * Don't forget to [enable two factor authentication](https://www.hackforla.org/guide-pages/2FA.html)  
+[See here](client/README.md) for more information about 311 Data client technology.
 
+### Server/API
 
-## 🎉🎉 After Signing up, Engineers start [here](https://github.com/hackforla/311-data/blob/master/GETTING_STARTED.md)!!!! 🎉🎉🎉
+* FastAPI: asynchronous Python REST API
+* Redis: key-value caching
+* PostgreSql: persistent SQL database
+* Prefect: data ingestion pipeline in Python
+* Docker: containerized servers hosted in AWS
 
-## Resources
-[Empower LA - HfLA Initial Meeting](https://docs.google.com/document/d/19jrYWjq_FfQbuqTfnwJFruWEo9pPF0R0qh4njDZsuzM)
+[See here](server/README.md) for more information about 311 Data server technology.
 
-[Empower LA - HfLA Initial Questions](https://docs.google.com/document/d/14WRgY_vjqG0FFLUPrB3Z4iARfm7cAsN3w0gjqdtoyjw/)
+### Data Analysis
 
-[Raw 311 data by year](https://data.lacity.org/browse?category=A+Well+Run+City&limitTo=filters&q=%22MyLA311+Service+Request+Data+201%22&sortBy=relevance)
+* Python
+* Pandas/Numpy/Matplotlib
+* PostgreSql
+* Socrata API
 
-## Existing 311 visualization tools:
-[Empower LA’s Demographics BI tool](https://empowerla.org/demographics-BI/)
+[See here](docs/data_loading.md) for more information about 311 Data project data analysis.
 
-[Mayor’s Dashboard](http://dashboard.lamayor.org/)
+### UI/UX
 
-[NYC 311 Data](http://people.ischool.berkeley.edu/~samuel.goodgame/311/)
+* Figma
+* Google Drive
+* Adobe CC
+* Miro
 
-[Neighborhood Council Funding System Dashboard](https://cityclerk.lacity.org/NCFundPortal/Dashboard.html)
+## Joining the 311 Data project
 
-## People involved in the project
-311-Data Roster: [Roster](https://docs.google.com/spreadsheets/d/1CZHH_91zTb9avfsJG9MtakCqbhLWQzTyTtQVNDqKqyM/edit#gid=0)
+We are looking for volunteers! Those with non-technical skills are welcome as well.
 
-Facilitator to Stakeholder: [EmpowerLA](http://empowerla.org/)
+311 Data is part of [Hack for LA](https://www.hackforla.org/) where you can sign up to join weekly onboarding meetings. You can find more information about the current project team [here](https://www.hackforla.org/projects/311-data).
 
-## Agendas
-[311-Data Agendas](https://docs.google.com/document/d/1Dr-AiOEBOGKDrAm7O2fxoZul2z5uSXH032QRzcUCYd0/edit)
+### To Sign Up
 
-## Working Documents
-[Diff Dashboard](https://docs.google.com/document/d/1CNEJ1yAa41WbjMLYDB-UuTUjnd51X5tTJ9kWVQlH9NM/edit)
+* Go to the [Getting Started](https://www.hackforla.org/getting-started) page on the Hack for LA website and learn about the projects and onboarding process.
+* Sign up for an onboarding session on the [Hack for LA Meetup](https://www.meetup.com/hackforla/events) page. These session happen every week!
 
-## Deliverables
-[Email to NC Members](https://docs.google.com/document/d/12JQ46SVsyywmdwEFPpE-Q1xFHdXNIP-AnbSaPKs0Kvk/edit#)
+## More Information about the 311 Data project
 
-[NC initial survey](https://drive.google.com/open?id=1N_cY23y4u04oHOlkyQId-K3k11J23lUGwMljmNHpmMk)
+* [Project background and resources](docs/background.md)
+* [Contributing to the project](docs/contributing.md)
+* [Data loading and analysis](docs/data_loading.md)

docs/background.md

@@ -0,0 +1,31 @@
+# Project Background and Resources
+
+The following resources provide good context for the project.
+
+## [LA 311 System Online](https://myla311.lacity.org/)
+
+The web version of the 311 system. For Los Angeles-based team members a good introduction to the project would be to create a service request ticket (there is likely graffiti or trash somewhere that you can report) and get first-hand knowledge of the system.
+
+## [EmpowerLA](http://empowerla.org/)
+
+The Los Angeles city department focused on community engagement and technology adoption.
+
+[Empower LA - HfLA Initial Meeting](https://docs.google.com/document/d/19jrYWjq_FfQbuqTfnwJFruWEo9pPF0R0qh4njDZsuzM)
+
+[Empower LA - HfLA Initial Questions](https://docs.google.com/document/d/14WRgY_vjqG0FFLUPrB3Z4iARfm7cAsN3w0gjqdtoyjw/)
+
+## [LA Open Data Portal](https://data.lacity.org/)
+
+Also called Socrata. This is the main source data system for the 311 Data project.
+
+[LA 2021 311 Service Request Data](https://data.lacity.org/City-Infrastructure-Service-Requests/MyLA311-Service-Request-Data-2021/97z7-y5bt): One year of data of 311 service requests. Includes a data dictionary of fields.
+
+[Geo Data for LA Neighborhoods](https://geohub.lacity.org/datasets/neighborhood-councils-certified): GeoJSON boundaries for the 99 official LA neighborhoods.
+
+## Other LA and 311 data reporting systems
+
+[Empower LA’s Demographics BI tool](https://empowerla.org/demographics-BI/)
+
+[LA Mayor’s Dashboard](http://dashboard.lamayor.org/)
+
+[Neighborhood Council Funding System Dashboard](https://cityclerk.lacity.org/NCFundPortal/Dashboard.html)

docs/contributing.md

@@ -0,0 +1,52 @@
+# Contributing to the project
+
+Welcome! This readme assumes you have already listened to the 311-data pitch, and gone through the basic onboarding. The following will be more geared towards the programming side of 311-data and getting your development environment setup. If you run into any problems, please submit a new issue.
+
+## Client Development
+
+For JavaScript developers looking to contribute to the 311 Data client, start by reading [how to set up a local client environment](../client/README.md)
+
+## API Server Development
+
+For Python developers looking to contribute to the 311 Data server, start by reading [how to set up a local API server](./server_setup.md)
+
+## Technology and system architecture
+
+Here is a high-level look at our architecture:
+![System diagram](images/data-arch.png)
+
+## Branching and Release Process
+
+For context on how we do branches, start by reviewing the [311 Data release process](./upgrades.md)
+
+### Feature Branching
+
+For development we use feature branching to ensure easy collaboration. There aren't any rules to branch naming or how many branches you are allowed to have, but the recommended convention would look like `issueId-Prefix-MinimalDescription`
+For example, a documentation branch could look like `138-DOC-OnboardingUpdate`.
+
+Read more about feature branching [here](https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow).
+
+### Default Branch
+
+Building on feature branching, we treat the `dev` branch as the main contribution branch. Pull requests to this branch should be as frequent as developers are closing issues *(Hopefully very frequent!)*. Pushes to `master` will be much less frequent and will be handled by administrators. With this workflow, `master` will have an extra layer of protection and should always represent a working version of the application.
+
+In other words, whenever you are about to start on a new feature, checkout your branch based off of the `dev` branch. Your command would look something like `git checkout -b 567-BACK-NewEndpoint dev`. See [this stackoverflow post](https://stackoverflow.com/questions/4470523/create-a-branch-in-git-from-another-branch) for more context.
+
+### Branch Protection/Github Actions
+
+We use [Github Actions](https://github.com/features/actions) to run our continuous integration (CI). These actions include status checks that run whenever you submit a pull request to `dev` or `master`. When you submit a PR, Github will run a set of operations to build and test all or part of the codebase. If any of these steps fail, the pull request will not be allowed to be merged until they are fixed. From the pull request UI you can find the reason an operation may have failed in the status checks section towards the bottom.
+
+If you want to look at our setup, check out the "Actions" tab in Github, as well as the [workflows directory](https://github.com/hackforla/311-data/tree/master/.github/workflows), which contains the code that Github runs when actions are triggered.
+
+In addition to status checks, PR's are required to have at least one reviewer before being merged into `dev` or `master`.
+
+## Testing
+
+CI Is driven by tests, they help instill confidence in pull requests because a developer can say "All the status checks pass and my new tests pass so the PR is safe to merge" When contributing new features, it is most ideal to write at least 4 tests targeting your code.
+
+- One for the "happy path": Test the endpoint/feature in the way it is intended to be used
+- One for the "extreme path": Test with extreme inputs/characteristics (What if I use 10,000 XYZ)
+- One for the "negative path": Test with strange input, (What if I send characters to a function that expects integers)
+- One for the "null path": Test with empty params/nothing/emptiness
+
+Our front end tests are run through Cypress and our backend tests are run through Pytest.

server/docs/data_fields.md → docs/data_loading.md

@@ -1,15 +1,14 @@
-# Data Fields
+# Data Loading
 
-## Notes about 311 Data
+Data is pulled nightly from the [LA Open Data](https://data.lacity.org/) website using a [Prefect](https://www.prefect.io/) workflow. The workflow loads new records and updates any that have been modified. Prefect tasks then update the API cache and reloads the dashboards.
 
-1. Not all requests are associated with a neighborhood council. There are some areas such as Pacific Palisades, Brentwood, and parts of South LA that are not covered by councils. Therefore there will always be more requests at the city-wide level than the sum of the council.
-2. The Report Water Waste request type is currently being ignored.
+![Data Loading Process](images/data-loading.png)
 
-## TABLE: requests
+## LA Open Data Format
 
-Contains 311 Service Request data for 2019 and 2020.
+The service request data from the LA Open Data system is divided up into different data source for each year but they all follow essentially the same format.
 
-Source is here and includes data dictionary:
+A sample data source is here and includes the data dictionary:
 https://data.lacity.org/A-Well-Run-City/MyLA311-Service-Request-Data-2020/rq3b-xjk8
 
 Fields that are pulled into the 311 Data system a listed below.
@@ -143,3 +142,17 @@ Fields that are pulled into the 311 Data system a listed below.
 ### latitude (double)
 
 ### longitude (double)
+
+## Notes about 311 Data
+
+1. Data is loaded with very little transformation. It is formatted for reporting by joining with clean dimensions.
+2. Not all requests are associated with a neighborhood council. There are some areas such as Pacific Palisades, Brentwood, and parts of South LA that are not covered by councils. Therefore there will always be more requests at the city-wide level than the sum of the council.
+3. The Report Water Waste request type is currently being ignored.
+
+## Other Open Data sources
+
+While the 311 data sources are the primary ones, we also load geographic data for the councils. Demographic and safety data are also available.
+
+* [Neighborhood Geographic data](https://geohub.lacity.org/datasets/neighborhood-council-boundaries-2018/data?geometry=-121.842%2C33.513%2C-115.019%2C35.101)
+* [Neighborhood Demographic data](https://geohub.lacity.org/datasets/demographics-of-neighborhood-councils?geometry=-118.566%2C34.016%2C-118.140%2C34.116)
+* [Citywide Crime data](https://data.lacity.org/Public-Safety/Crime-Data-from-2020-to-Present/2nrs-mtv8)