ingestion of all data #479
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sellnat77
reviewed
Mar 31, 2020
sellnat77
reviewed
Mar 31, 2020
sellnat77
reviewed
Mar 31, 2020
server/src/services/sqlIngest.py
Outdated
| async def populateDatabase(self, | ||
| years=range(2015, 2021), | ||
| limit=2000000, | ||
| querySize=50000): |
There was a problem hiding this comment.
We might want to move these default numbers to config that way theres a single source of truth
There was a problem hiding this comment.
I put the defaults in the config, but I consumed the defaults in app.py instead of here because those default values were also appearing in app.py. So the defaults get applied immediately if the query params are missing, and then passed into this service.
|
Fixed that stuff. I also changed /ingest to a GET request because we're using query params for it instead of the body. Makes sense yeah? |
sellnat77
approved these changes
Mar 31, 2020
This was referenced Apr 1, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The goal here was to make sure it was possible to get all of the socrata data -- from 2015 through today -- into our own database. This code did that successfully, in a mere 74 minutes :)
I used sqlalchemy instead of pandas to create the table and insert and all the data. This made it possible to eliminate several of the cleaning tasks we were doing previously, such as converting columns to datetime and making sure the pandas columns were in sync with the database model. It also makes the Ingest model the single point of truth for the structure of the table.
There was one tricky issue, discussed on Slack, regarding the quirky sorting coming from the socrata api. I eliminated the sort parameter, as Russell suggested, but was still getting intermittent primary key violations due to duplicate srnumbers in separate batches from socrata. So what I did was this:
create a temporary id column on the Ingest model to use as the primary key
after all the data is ingested, deduplicate the table by
srnumber, delete the temporary id column, and make thesrnumberthe primary key. Not pretty, but it works.Other minor things:
I put the socrata client into a separate class that handles automatic retries in the event that socrata times out (which happens pretty often)
added a cleaning task that converts the
closseddatevalues to null if they precede thecreateddate(fixing closed date is sometimes before created date #456)