This repository has been archived by the owner on Sep 10, 2020. It is now read-only.
Populate raw schema in development VM #89
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
conorsch
reviewed
Nov 30, 2016
| always_run: true | ||
| changed_when: false | ||
|
|
||
| - debug: var=raw_schema_population_result.results |
There was a problem hiding this comment.
Do you still want these debug line in here? FYI recent versions of Ansible support the verbosity parameter on debug tasks, so the associated message will only display with e.g. -vvv if verbosity: 3. http://docs.ansible.com/debug_module.html
There was a problem hiding this comment.
Hmm okay thanks for pointing that out, it's not necessary so I will remove this real quick
There was a problem hiding this comment.
Snipped that line out
redshiftzero
force-pushed
the
populate-raw-schema-in-vm
branch
from
5a73245 to
d8c2c2d
Compare
|
Changes Unknown when pulling d8c2c2d on populate-raw-schema-in-vm into ** on master**. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the population of the raw schema to the Ansible playbook that provisions the development VM. This is done such that the feature generation and machine learning codes can be run more easily in the development VM - i.e. without having to run the crawler (can take a while) or connect to the production database (A Bad Idea). The data that is populating the raw schema in the VM is derived from our real data (with some anonymization). I also add the notebook where I construct this dataset for future reference / modification.
Upon request, I have also created a version of the data used to populate each individual table here for people to play with in a single file
roles/crawler/files/raw-data/test_data.csvwithout needing to worry about joins.This PR also bumps the version of Tor Browser since our download link in the Ansible play was old and the download link was 404ing