Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating master with latest dev #244

Merged
merged 75 commits into from
Feb 12, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
e539a76
Restructed folder layout and html layout
Jan 22, 2020
9162ea4
Changed databaseOrm to be a relative import in sqlIngest.py
Jan 28, 2020
81dc730
Added classnames dependency. Used for dynamic className generation to…
adamkendis Jan 28, 2020
89ea9ca
Generic button component adhering to Bulma styling modifiers.
adamkendis Jan 28, 2020
b0c5060
Rephrased comment for greater clarity.
adamkendis Jan 28, 2020
99de5cb
Added bulma-checkradio and react-id-generator dependencies.
adamkendis Jan 28, 2020
171e794
Unique id generation added to Button component.
adamkendis Jan 28, 2020
79ffb1c
Bulma-checkradio import.
adamkendis Jan 28, 2020
08fd652
Generic checkbox component adhering to Bulma-checkradio extension sty…
adamkendis Jan 28, 2020
7e6bc5b
Added type attribute to Checkbox props.
adamkendis Jan 28, 2020
0cce2f3
Added bulma-switch dependency and import.
adamkendis Jan 28, 2020
e3a456d
Fixed Checkbox defaultProps.
adamkendis Jan 28, 2020
a19c901
Generic toggleswitch component adhering to Bulma-switch extension sty…
adamkendis Jan 28, 2020
e832473
Added leftLabel and rightLabel to ToggleSwitch props.
adamkendis Jan 28, 2020
976ecc4
Merge remote-tracking branch 'upstream/dev' into dev
rgao Jan 29, 2020
fd76ed3
Wrote /ingest API endpoint to pull data from Sanic into Postgres DB, …
Jan 29, 2020
8822681
Removed unnecessary line.
Jan 29, 2020
a5e5b35
Added more robust error-handling.
Jan 29, 2020
04a9964
Better error handling.
Jan 29, 2020
87f1e05
Fixed typo.'
Jan 29, 2020
e90fce9
Fixed lint error.
Jan 29, 2020
67cf403
Removed trailing whitespace lint error.
Jan 29, 2020
46bed62
Added quickview
Jan 29, 2020
49e8403
Added id field to props and removed react-id-generator.
adamkendis Jan 29, 2020
5b69eeb
Sanic control point for database ingestion
ryanmswan Jan 29, 2020
f676b98
Added colors constant file
Jan 29, 2020
74e8200
changed connection string to mysql
rgao Jan 29, 2020
362b6b7
fixing merging
rgao Jan 29, 2020
9702b3b
changed connection string to using mysql in example config, added var…
rgao Jan 30, 2020
937ccc6
actually added connection string changes to commit this time
rgao Jan 30, 2020
fb0e85e
removed excess parameter values in sqlalchemy ingestion, added commen…
rgao Jan 30, 2020
cbfa24a
fixed linting error, updated varchar specifications, docstrings now r…
rgao Jan 30, 2020
d24e2d8
added dialect as a class property
rgao Jan 30, 2020
086a9c3
Removed react-id-generator from dependencies.
adamkendis Jan 30, 2020
dbb98a5
Merge branch 'dev' into 180_FRONT_genericComponents
adamkendis Jan 30, 2020
0540eb8
updated varchar specifications for full data ingestion
rgao Jan 31, 2020
e1acfdc
Sidebar testing
brodly Feb 1, 2020
089f7af
Merge pull request #223 from rgao/dev
ryanmswan Feb 1, 2020
d632088
Changed id to required prop. Other minor tweaks to props and comments.
adamkendis Feb 1, 2020
7c7658b
Merge branch 'dev' into 180_FRONT_genericComponents
adamkendis Feb 1, 2020
b06624c
Merge branch 'dev' into 180_FRONT_genericComponents
adamkendis Feb 1, 2020
6c5f7d3
Merge branch 'dev' into 180_FRONT_genericComponents.
adamkendis Feb 1, 2020
33f26ff
Merge pull request #220 from adamkendis/180_FRONT_genericComponents
brodly Feb 1, 2020
885f19b
Footer sticks to bottom
brodly Feb 1, 2020
8a1ea56
Removed unused components
brodly Feb 1, 2020
5a1cbd3
Testing sidebar menu
brodly Feb 1, 2020
7295013
Merge branch 'dev' of https://github.com/hackforla/311-data into rest…
brodly Feb 1, 2020
b126aa3
Bring fork up to date with dev
ryanmswan Feb 1, 2020
72d73db
Added cleaner object
ryanmswan Feb 1, 2020
e56ba94
Merge pull request #224 from hackforla/restructure_application
adamkendis Feb 5, 2020
7816a12
Merge branch 'dev' into dev
ryanmswan Feb 5, 2020
92ec984
Merge pull request #227 from ryanmswan/dev
ryanmswan Feb 5, 2020
6609f09
Removed react-sidebar. Imported react-burger-menu
brodly Feb 5, 2020
72804d6
Added style props
brodly Feb 5, 2020
7b45fc8
Added id for menu slide
brodly Feb 5, 2020
e7949ad
Renamed footer and header elements
brodly Feb 5, 2020
ef8af97
Sidebar slides in and out with button toggle
brodly Feb 5, 2020
bda73f6
Moved menu to its own folder. Added tab interactions
brodly Feb 5, 2020
a57ca93
Merge branch 'dev' of https://github.com/hackforla/311-data into 225_…
brodly Feb 5, 2020
763b7a0
Merge pull request #233 from hackforla/225_sidebar
adamkendis Feb 6, 2020
e5c7ff1
Dropdown component in progress.
adamkendis Feb 7, 2020
77054f7
Added react-hooks plugin and rules to eslint.
adamkendis Feb 8, 2020
4d7ea33
Dropdown closing on outside click implemented.
adamkendis Feb 8, 2020
5c39348
Dropdown and DropdownItem mostly working.
adamkendis Feb 11, 2020
50d3a7f
Comments and code cleanup.
adamkendis Feb 11, 2020
eb22a6e
Moved DropdownItem component into separate file.
adamkendis Feb 11, 2020
84736d5
Dropdown styles and code cleanup.
adamkendis Feb 11, 2020
09934f5
Changed default width.
adamkendis Feb 11, 2020
e1ecbe2
Added checked prop to Checkbox.
adamkendis Feb 11, 2020
bacaaa8
Added width prop to Dropdown and DropdownItem.
adamkendis Feb 11, 2020
c5652fa
Merge pull request #243 from adamkendis/166_FRONT_dateFilter
brodly Feb 11, 2020
7810808
Fixes #235 Enforced a frontend linting step to protect PRs
sellnat77 Feb 9, 2020
323d07e
Merge pull request #241 from hackforla/235_FE_linting
sellnat77 Feb 12, 2020
bbc72dd
Merge branch 'master' of github.com:hackforla/311-data into UpdateDev
sellnat77 Feb 12, 2020
ecabb53
Merge pull request #245 from hackforla/UpdateDev
ryanmswan Feb 12, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
*.json
*.test.js*
3 changes: 3 additions & 0 deletions .eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,11 @@ module.exports = {
},
plugins: [
'react',
'react-hooks'
],
rules: {
'linebreak-style': 'off',
"react-hooks/rules-of-hooks": "error",
"react-hooks/exhaustive-deps": "warn"
},
};
2 changes: 2 additions & 0 deletions .github/workflows/Continuous_Integration_Frontend.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ jobs:
node-version: ${{ matrix.node-version }}
- name: Install Packages
run: npm install
- name: Lint
run: npm run lint
- name: Build project
run: npm run build
- name: Run Tests
Expand Down
7 changes: 6 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@
"axios": "^0.19.0",
"babel-jest": "^24.9.0",
"bulma": "^0.8.0",
"bulma-checkradio": "^1.1.1",
"bulma-switch": "^2.0.0",
"classnames": "^2.2.6",
"dataframe-js": "^1.4.3",
"dotenv-webpack": "^1.7.0",
"gh-pages": "^2.1.1",
Expand All @@ -14,6 +17,7 @@
"leaflet": "^1.5.1",
"proptypes": "^1.1.0",
"react": "^16.8.6",
"react-burger-menu": "^2.6.13",
"react-dom": "^16.8.6",
"react-leaflet": "^2.4.0",
"react-leaflet-choropleth": "^2.0.0",
Expand All @@ -29,7 +33,8 @@
"start": "npm run dev",
"dev": "webpack-dev-server --config webpack.dev.js --host 0.0.0.0",
"build": "webpack --config webpack.prod.js",
"test": "jest",
"lint": "eslint src/**/*.js*",
"test": "jest --passWithNoTests",
"predeploy": "npm run build",
"deploy": "gh-pages -d dist"
},
Expand Down
2 changes: 1 addition & 1 deletion public/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
crossorigin=""/>
<title>311 Data</title>
</head>
<body>
<body class="has-navbar-fixed-bottom">
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<script src="bundle.js"></script>
Expand Down
27 changes: 17 additions & 10 deletions server/src/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
from configparser import ConfigParser
from threading import Timer
from multiprocessing import cpu_count
from services.sqlIngest import DataHandler
from datetime import datetime


app = Sanic(__name__)
Expand Down Expand Up @@ -64,17 +66,22 @@ async def sample_route(request):

@app.route('/ingest', methods=["POST"])
async def ingest(request):
'''Accept POST requests with a list of datasets to import\
based on the YearMapping. Body parameter format is \
{"sets": ["YearMappingKey","YearMappingKey","YearMappingKey"]}'''

ingress_worker = ingress_service(config=app.config['Settings'])
"""Accept POST requests with a list of years to import.
Query parameter name is 'years', and parameter value is
a comma-separated list of years to import.
Ex. '/ingest?years=2015,2016,2017'
"""
current_year = datetime.now().year
ALLOWED_YEARS = [year for year in range(2015, current_year+1)]
if not request.args.get("years"):
return json({"error": "'years' parameter is required."})
years = set([int(year) for year in request.args.get("years").split(",")])
if not all(year in ALLOWED_YEARS for year in years):
return json({"error": f"'years' parameter values must be one of {ALLOWED_YEARS}"})
loader = DataHandler()
loader.loadConfig(configFilePath='./settings.cfg')
loader.populateFullDatabase(yearRange=years)
return_data = {'response': 'ingest ok'}

for dataSet in request.json.get("sets", None):
target_data = app.config["Settings"]["YearMapping"][dataSet]
return_data = await ingress_worker.ingest(from_dataset=target_data)

return json(return_data)


Expand Down
49 changes: 49 additions & 0 deletions server/src/services/dataCleaner.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
import sqlalchemy as db
import pandas as pd
from sqlIngest import DataHandler
import databaseOrm


class DataCleaner(DataHandler):
def __init__(self, config=None, configFilePath=None, separator=','):
self.data = None
self.config = config
self.dbString = None if not self.config \
else self.config['Database']['DB_CONNECTION_STRING']
self.filePath = None
self.configFilePath = configFilePath
self.separator = separator
self.fields = databaseOrm.tableFields
self.insertParams = databaseOrm.insertFields
self.readParams = databaseOrm.readFields

def fetchData(self):
'''Retrieve data from mySql database instance'''
engine = db.create_engine(self.dbString)
self.data = pd.read_sql('ingest_staging_table',
con=engine,
index_col='srnumber')

def formatData(self):
'''Perform changes to data formatting to ensure compatibility
with cleaning and frontend processes'''
pass

def groupData(self):
'''Cluster data by geographic area to remove repeat instances
of 311 reports'''
pass

def cleaningReport(self):
'''Write out cleaning report summarizing operations performed
on data as well as data characteristics'''
pass


if __name__ == "__main__":
'''Class DataHandler workflow from initial load to SQL population'''
cleaner = DataCleaner()
cleaner.loadConfig(configFilePath='../settings.cfg')
cleaner.fetchData()
# can use inherited ingestData method to write to table
cleaner.ingestData(tableName='clean_data')
50 changes: 25 additions & 25 deletions server/src/services/databaseOrm.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,40 +43,40 @@ class Ingest(Base):
policeprecinct = Column(String)


insertFields = {'srnumber': String,
insertFields = {'srnumber': String(50),
'createddate': DateTime,
'updateddate': DateTime,
'actiontaken': String,
'owner': String,
'requesttype': String,
'status': String,
'requestsource': String,
'createdbyuserorganization': String,
'mobileos': String,
'anonymous': String,
'assignto': String,
'servicedate': String,
'closeddate': String,
'addressverified': String,
'approximateaddress': String,
'address': String,
'housenumber': String,
'direction': String,
'streetname': String,
'suffix': String,
'actiontaken': String(30),
'owner': String(10),
'requesttype': String(30),
'status': String(20),
'requestsource': String(30),
'createdbyuserorganization': String(16),
'mobileos': String(10),
'anonymous': String(10),
'assignto': String(20),
'servicedate': String(30),
'closeddate': String(30),
'addressverified': String(16),
'approximateaddress': String(20),
'address': String(100),
'housenumber': String(10),
'direction': String(10),
'streetname': String(50),
'suffix': String(10),
'zipcode': Integer,
'latitude': Float,
'longitude': Float,
'location': String,
'location': String(100),
'tbmpage': Integer,
'tbmcolumn': String,
'tbmcolumn': String(10),
'tbmrow': Float,
'apc': String,
'apc': String(30),
'cd': Float,
'cdmember': String,
'cdmember': String(30),
'nc': Float,
'ncname': String,
'policeprecinct': String}
'ncname': String(100),
'policeprecinct': String(30)}


readFields = {'SRNumber': str,
Expand Down
25 changes: 13 additions & 12 deletions server/src/services/sqlIngest.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ def __init__(self, config=None, configFilePath=None, separator=','):
self.fields = databaseOrm.tableFields
self.insertParams = databaseOrm.insertFields
self.readParams = databaseOrm.readFields
self.dialect = None

def loadConfig(self, configFilePath):
'''Load and parse config data'''
Expand All @@ -33,6 +34,7 @@ def loadConfig(self, configFilePath):
config.read(configFilePath)
self.config = config
self.dbString = config['Database']['DB_CONNECTION_STRING']
self.dialect = self.dbString.split(':')[0]
self.token = None if config['Socrata']['TOKEN'] == 'None' \
else config['Socrata']['TOKEN']

Expand Down Expand Up @@ -82,21 +84,25 @@ def cleanData(self):
print('\tCleaning Complete: %.1f minutes' %
self.elapsedTimer(cleanTimer))

def ingestData(self, ingestMethod='replace'):
def ingestData(self, ingestMethod='replace',
tableName='ingest_staging_table'):
'''Set up connection to database'''
print('Inserting data into Postgres instance...')
asdf = 'Inserting data into ' + self.dialect + ' instance...'
print(asdf)
ingestTimer = time.time()
data = self.data.copy() # shard deepcopy for other endpoint operations
engine = db.create_engine(self.dbString)
newColumns = [column.replace(' ', '_').lower() for column in data]
data.columns = newColumns
# Ingest data
data.to_sql("ingest_staging_table",
# Schema is same as database in MySQL;
# schema here is set to db name in connection string
data.to_sql(tableName,
engine,
if_exists=ingestMethod,
schema='public',
index=False,
chunksize=10000,
chunksize=10,
dtype=self.insertParams)
print('\tIngest Complete: %.1f minutes' %
self.elapsedTimer(ingestTimer))
Expand Down Expand Up @@ -171,7 +177,7 @@ def populateFullDatabase(self, yearRange=range(2015, 2021)):
Default operation is to fetch data from 2015-2020
!!! Be aware that each fresh import will wipe the
existing staging table'''
print('Performing fresh Postgres population from Socrata data sources')
print('Performing fresh ' + self.dialect + ' population from Socrata data sources')
tableInit = False
globalTimer = time.time()
for y in yearRange:
Expand Down Expand Up @@ -238,11 +244,6 @@ def fix_nan_vals(resultDict):
'''Class DataHandler workflow from initial load to SQL population'''
loader = DataHandler()
loader.loadConfig(configFilePath='../settings.cfg')
loader.fetchSocrataFull(limit=10000)
loader.fetchSocrataFull()
loader.cleanData()
loader.ingestData()
loader.saveCsvFile('testfile.csv')
loader.dumpFilteredCsvFile(dataset="",
startDate='2018-05-01',
requestType='Bulky Items',
councilName='VOICES OF 90037')
loader.ingestData('ingest_staging_table')
2 changes: 1 addition & 1 deletion server/src/settings.example.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ HOST = 0.0.0.0
PORT = 5000

[Database]
DB_CONNECTION_STRING = postgres://REDACTED:REDACTED@localhost:5432/postgres
DB_CONNECTION_STRING = mysql://REDACTED:REDACTED@localhost:5432/public
DATA_DIRECTORY = static

[Api]
Expand Down
95 changes: 17 additions & 78 deletions src/App.jsx
Original file line number Diff line number Diff line change
@@ -1,83 +1,22 @@
import React, { Component } from 'react';
import axios from 'axios';

import { getDataResources } from './Util/DataService';
import { REQUESTS, COUNCILS } from './components/common/CONSTANTS';
import React, { useEffect } from 'react';
import { connect } from 'react-redux';

import Header from './components/main/header/Header';
import Body from './components/main/body/Body';
import Footer from './components/main/footer/Footer';

class App extends Component {
constructor() {
super();

this.state = {
data: [],
year: '2015',
startMonth: '1',
endMonth: '12',
request: REQUESTS[0],
showMarkers: false,
showMarkersDropdown: true,
};
}

componentDidMount() {
this.fetchData();
}

updateState = (key, value, cb = () => null) => {
this.setState({ [key]: value }, () => {
this.fetchData(); // This is only for the dropdown component to fetch data on change
cb();
});
}

toggleShowMarkers = () => {
const { showMarkers } = this.state;
this.setState({ showMarkers: !showMarkers });
}

fetchData = () => {
const dataUrl = this.buildDataUrl();

axios.get(dataUrl)
.then(({ data }) => {
this.setState({ data });
})
.catch((error) => {
console.error(error);
});
}

buildDataUrl = () => {
const {
startMonth, endMonth, year, request,
} = this.state;
const dataResources = getDataResources();
return `https://data.lacity.org/resource/${dataResources[year]}.json?$select=location,zipcode,address,requesttype,status,ncname,streetname,housenumber&$where=date_extract_m(CreatedDate)+between+${startMonth}+and+${endMonth}+and+requesttype='${request}'`;
}

render() {
const { data, showMarkers, showMarkersDropdown } = this.state;

return (
<div className="main">
<Header
updateState={this.updateState}
toggleShowMarkers={this.toggleShowMarkers}
showMarkers={showMarkers}
showMarkersDropdown={showMarkersDropdown}
/>
<Body
data={data}
showMarkers={showMarkers}
/>
<Footer />
</div>
);
}
}

export default App;
const App = () => {
useEffect(() => {
// fetch data on load??
}, []);

return (
<div className="main">
<Header />
<Body />
<Footer />
</div>
);
};

export default connect(null, null)(App);
Loading