README.md

Jobs for Hope

Aggregate job opportunities with homeless service providers so that people can search for a job that fits their skillset.

Prerequisites

1. Git for working with Github source code
2. Node and npm for running the web app
3. Python 2 and pip for running the web scraper

Complete beginner instructions

click here to show

Windows

show

1. Install chocolatey (https://chocolatey.org/install)
   1. Install chocolatey gui (optional)

      choco install chocolateygui -y
      

2. Install git, chromium, chromedriver, vscode, python2, dbeaver, nodejs

   choco install git chromium chromedriver vscode python2 dbeaver nodejs -y
   

3. Install postgresql and set postgres user password

   choco install postgresql --params '/Password:password' --params-global -y
   

Windows 10

show

1. Enable Windows Subsystem for Linux (reference: https://docs.microsoft.com/en-us/windows/wsl/install-win10 https://lifehacker.com/how-to-get-started-with-the-windows-sybsystem-for-linux-1828952698)
   1. In the search bar, type "turn windows features on or off" and choose the correct item
   2. Scroll down and check the box for Windows Subsystem for Linux
   3. Windows will restart to complete the installation
2. Install Ubuntu Linux
   1. Open the Microsoft Store and search for "Run Linux on Windows"
   2. Install and launch Ubuntu
   3. Set up a new linux user account when running for the first time
   4. Update and upgrade all packages
      1. In a terminal, run (you will need to type in your user password when running sudo)

         sudo apt update && sudo apt upgrade -y
         

3. Install ChomeDriver
   1. Install chocolatey (https://chocolatey.org/install)
   2. Install ChomeDriver

      choco install chromedriver -y
      

   3. Add a file chromedriver to the project directory with this content

      #!/bin/sh
      chromedriver.exe "$@"
      

4. Continue to Linux instructions

Linux

show

1. Install Homebrew on Linux (reference: https://docs.brew.sh/Homebrew-on-Linux)
   1. Open a Linux terminal
   2. Install dependencies
      • Debian-based (Ubuntu)

        sudo apt install build-essential curl file git
        

      • Fedora-based

        sudo yum groupinstall 'Development Tools' && sudo yum install curl file git
        

   3. Install homebrew

      sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"
      

2. Install packages brew install python@2 postgresql sudo apt install chromium-chromedriver -y

macOS

show

1. Install homebrew (https://brew.sh/)
2. Install packages brew install git python@2 postgresql brew cask install chromedriver

Common Tools for all OSes

show

1. Install Visual Studios Code (https://code.visualstudio.com/)

   1. Install Prettier - Code formatter extension

2. Install DBeaver (https://dbeaver.io/), Community Edition

3. Install nvm, node, and npm (reference: https://gist.github.com/d2s/372b5943bce17b964a79)

   1. Install nvm

      curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash
      

   2. Install the latest LTS Node.js (https://nodejs.org/en/)

      nvm install v10.15.3
      

4. Download and install Chrome (https://www.google.com/chrome/)

5. Setup Postgresql (reference: https://github.com/michaeltreat/Windows-Subsystem-For-Linux-Setup-Guide/blob/master/readmes/installs/PostgreSQL.md)

   1. Start the postgres service

      1. Start a terminal app
         1. linux (and WSL)
            1. Start the service

               sudo service postgresql start
               

            2. Setup the postgres user (linux)
               1. Start a terminal app
               2. Set the password

                  sudo passwd postgres
                  

               3. Type in the password and confirmation
               4. Close the terminal
            3. Connect to postgres
               1. Start a terminal app
               2. Switch to the postgres user and start the psql prompt

                  sudo -u postgres psql
                  

               3. If the above doesn't work, do this instead

                  su - postgres
                  psql
                  

            4. Troubleshooting postgres on WSL (reference: microsoft/WSL#3863)
               1. Append this at the end of /etc/postgresql/10/main/postgresql.conf

                  data_sync_retry = true
                  

         2. macOS
            1. Start the service

               brew services start postgresql
               

            2. Connect to postgres
               1. Start a terminal app
               2. enter the psql prompt

                  psql postgres
                  

   2. Create the database (reference: https://www.techrepublic.com/blog/diy-it-guy/diy-a-postgresql-database-server-setup-anyone-can-handle/)

      1. Start the psql prompt
      2. Issue the command

         create database jobsforhope;
         

      3. Create the user
         1. Start the psql prompt
         2. Issue the command

            create user jobsforhope;
            

         3. Check that the user was created

            \du
            

      4. Grant user privilege 1. Start the psql prompt 1. Issue the command grant all privileges on database jobsforhope to jobsforhope;

Full-Stack React/Node Application Installation

1. Clone this repo to your local drive.

   details

   1. Start a terminal app, such as Ubuntu for Windows Subsystem for Linux
   2. Create a src directory in the user's home directory and go in it

      cd && mkdir src && cd src
      

   3. Clone the repository

      git clone https://github.com/hackforla/jobs-for-hope
      

2. Change to the jobs-for-hope directory:

   cd jobs-for-hope
   

3. Install the node server npm depedencies:

   npm install
   

4. Obtain the .env file from the slack channel and place it in this directory. It contains private info (i.e., the production database connection string) that we cannot put in this public GitHub repo.

5. Change to the client directory:

   cd client
   

6. Install the client (React) dependencies:

   npm install
   

To Run the React/Node Application

1. Run npm start from the jobs-for-hope directory to start the node server.
2. Run npm start from the jobs-for-hope/client directory to start the react app and open the browser.

Backend Python Scraper Setup

1. Virtualenv (optional, but will be useful if you eventually want to have multiple python projects)

   show

   1. Set up virtualenv

      1. Install virtualenv using pip

         pip install virtualenv virtualenvwrapper
         

      2. cd into project directory

         cd jobs-for-hope/backend
         

      3. create the virtualenv

         virtualenv venv
         

         Alternative for systems where python 2.7 is not the default

         1. Specify the python location when creating the virtualenv

            virtualenv -p /usr/local/bin/path/to/python2.7 venv # use this if the system default is python3
            

      alternative setup using virtualenvwrapper

      1. Install virtualenv using pip

         pip install virtualenv virtualenvwrapper
         

      2. Create directory to hold virtual environments

         mkdir $HOME/.virtualenvs
         

      3. Find out where virtualenvwrapper.sh is located for next step

         which virtualenvwrapper.sh
         

      4. Make .bash_profile call .bashrc (reference:http://www.joshstaiger.org/archives/2005/07/bash_profile_vs.html)

         1. Add this to .bash_profile

            if [ -f ~/.bashrc ]; then
              source ~/.bashrc
            fi
            

      5. Add to .bashrc

         export WORKON_HOME=$HOME/.virtualenvs
         export PATH="/path/to/virtualenvwrapper:$PATH"
         source virtualenvwrapper.sh
         

      6. Start a new terminal session or call .bashrc

         source ~/.bashrc        # activate virtualenvwrapper.sh, just for the first time
         

      7. Create the virtualenv

         mkvirtualenv jobs-for-hope
         

         Alternative for systems where python 2.7 is not the default

         1. Specify the python location when creating the virtualenv mkvirtualenv -p /usr/local/bin/path/to/python2.7 jobs-for-hope # use this if the system default is python3

   2. Activate the virtualenv

      source venv/bin/activate
      

      for virtualenvwrapper

      workon  // list the existing virtual environments, blank if none is created
      workon jobs-for-hope  // activate virtual environment, not needed when first creating the virtualenv
      

   3. Do work and run python within the virtualenv

   4. Deactivate the virtualenv

      deactivate // switch back to system python
      

   1. All python commands should be run inside the jobs-for-hope virtualenv if you choose to set it up. You will have to make sure the virtualenv is activated before following the steps.

2. Install project dependencies

   cd backend
   pip install -r ./requirements.txt
   

3. Setup for the scrapers

   1. Download the database.ini file from the slack channel into the backend directory
   2. Dump the aws database to a file (reference database.ini for values)

      pg_dump -d <database> -h <host> -U <user> -C -f jobsforhope.sql
      

   3. restore to the local database

      psql -h localhost -U postgres -d jobsforhope -f jobsforhope.sql
      

To run the scraper

1. Switch to the backend directory

   cd backend
   

2. To run all the scrapers

   python scraper_runner.py
   

3. To run a single scraper

   python scraper_runner.py scrapers/prototypes.py
   

Checking the database for scraped data

1. Use DBeaver
2. Add a postgres database connection with the .env file credentials from the slack channel.
3. The jobs are in the jobs table.