Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add endpoint to store scraped courses #47

Merged
merged 4 commits into from
Feb 24, 2020

Conversation

austinsilveria
Copy link
Collaborator

@austinsilveria austinsilveria commented Feb 20, 2020

What's New?

  • save_courses API accepts a JSON object containing a list of all scraped courses (request.form did not play nicely with lists)
  • courseId is now a string to uniquely identify each course (e.g. CSC 101)
  • validate_input_keys() pulled out to its own function to be reused

Fixes #46

Type of change (pick-one)

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Integration test with course scraper stores all scraped courses in local MySQL instance.

Steps

  1. Ensure your local connection to MySQL
  2. Clone https://github.com/calpoly-csai/csai-scraping
  3. Run flask_api.py in api package
  4. Run sustainer.py in csai-scraping package (optionally comment out other scrapers if you do not want them to run) (branch with updated csai-scraping)

Checklist (check-all-before-merge)

formatting help: - [x] means "checked' and - [ ] means "unchecked"

  • I documented my code according to the Google Python Style Guide

  • I ran ./build_docs.sh and the docs look fine

  • I ran ./type_check.sh and got no errors

  • I ran ./format.sh because it automatically cleans my code for me 😄

  • I ran ./lint.sh to check for what "format" missed

  • I added my tests to the /tests directory

  • I ran ./run_tests.sh and all the tests pass

database_wrapper.py Outdated Show resolved Hide resolved
This was referenced Feb 23, 2020
Entity/Courses.py Outdated Show resolved Hide resolved
@mfekadu mfekadu merged commit 9f76075 into calpoly-csai:dev Feb 24, 2020
@mfekadu
Copy link
Member

mfekadu commented Feb 24, 2020

It worked for me and the code is fantastic, thank you @austinsilveria !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add API endpoint to store scraped courses
3 participants