Scraper #316
Closed
Scraper #316
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mikeyavorsky
requested changes
Feb 21, 2024
| @@ -25,7 +25,7 @@ services: | |||
| api: | |||
| build: | |||
| context: . | |||
| dockerfile: ./backend/Dockerfile | |||
| dockerfile: ./backend/Dockerfile.cloud | |||
There was a problem hiding this comment.
This change should be reverted, breaks docker-compose build on ARM Macs.
|
Something to consider... we'll need to add some logic to handle 429 responses. Right now, it just continues making requests, but ideally, it should suspend the process and pick it up after a time. Even better it could track the occurrences of these requests and collect data on them for review. |
DMalone87
force-pushed
the
ryan/scraper-update
branch
from
1736fc6
to
90f9bb6
Compare
aliavni
reviewed
Apr 20, 2024
There was a problem hiding this comment.
Hi folks, I am a new contributor to this repo. This work looks nice. Here is my five cents:
- We may want to have scraping-related work outside this repo. We already have quite a bit going on here. Having a separate repo would be easier in the long for example for deployments and requirements management.
- I think we can eliminate the redis portion. Once scrapers generate data files, we can load the new data using upserts.
- There are Python packages dedicated to scraping. One of the great ones is Scrapy. It is a pretty mature package with all kinds of conveniences such as handling retries, user agents etc.
There was a problem hiding this comment.
Here is an example scraper with Scrapy I built before https://github.com/aliavni/drugbank-scraper
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Scraper Update 🕷️
Sorry for the big PR; got carried away having to convert the code from one repo to another. For simplicity, this scraper code runs asynchronously with a cronjob that calls the flask command
sracpe-v2. This iteratively crawls through the 50-a website, gathers as much information as it can (still working on agencies), and downloads the CSV's from NYPD. I then check if the information is in our scraping cache; if not, we add it to the database. The incident schemas are a bit whack and may create duplicated information based on whether the officer is a perpetrator, participant, etc., but they should get the job done for now.