Parse regulations on local

Summary

The eregs API and parsing functions reside in the same repository. As the repository has evolved, the programming language versions and their packages needed to support both functions have diverged.

A high-level skeleton for running the API and parsing the regulations locally is:

• Use two Python 3.9.x virtual environment terminals:

  • First for running the eregs API
  • Second for parsing the FEC's regulations

• Install two different set of requirements files:

  • requirements.txt for running the eregs API
  • requirements-parsing.txt for parsing the FEC's regulations

Setup and parse the latest 202x regulations on local environment

How to setup two terminals (eregs API and parser)

• Setup eregs API terminal:

  • Clone fec-eregs repo and create python virtualenv and

    git clone https://github.com/fecgov/fec-eregs.git
    pyenv virtualenv 3.9.x venv-eregs
    pyenv activate venv-eregs
    pip install -r requirements.txt
    

  • Create local_settings.py in fec-eregs repo (if one doesn't exist)

    echo API_BASE = 'http://localhost:8000/api/' >> local_settings.py
    

  • Copy the following database configuration onto the local_setting.py file

        DATABASES = {
          'default': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'NAME': 'eregs_local',
            'HOST': '127.0.0.1',
            'PORT': '5432',
          }
        }
    

  • Create eregs_local database (Database name should match with the NAME variable inside local_settings.py)

    dropdb eregs_local # if exist
    createdb eregs_local
    

  • Setup node environment

    nvm install v14.15.5
    nvm use 14.15.5
    nvm alias default 14.15.5
    npm install -g grunt-cli
    

  • Install dependencies from package.json

    rm -rf node_modules/ # if exist
    npm i
    npm run build
    

  • Apply django database migrations

    python manage.py migrate
    

  • Build and compile frontend assets

    python manage.py compile_frontend   
    

  • Start server

     python manage.py runserver (leave this running)
    

• Setup parser terminal:

  • In a new terminal navigate to fec-eregs repo. Create Python virtualenv 3.9.x and activate it.

    pyenv virtualenv 3.9.x eregs-parser
    pyenv activate eregs-parser
    pip install -r requirements-parsing.txt
    

  • Run parser which write 45 regulations to the eregs API. It will take ~10 minutes to finish parsing 45 regulations.

    python load_regs/load_fec_regs.py local
    

• Copy & paste below URL in a browser to view all 45 regulations.

  http://127.0.0.1:8000/
  

• Verify the following to make sure regulations are being written to local eregs API

  • The following message displays in eregs API terminal which should include 204 successful code.
    e.g. "/api/layer/formatting/cfr/2021-annual-1/1 HTTP/1.1" 204 0..."

  • The following message should appear in the parser terminal, after the last regulation 9039 parses successfully.
    e.g. "Export output - 11 CFR 9039, Destination: http://localhost:8000/api..."

  • Check the following tables in eregs_local database.
    e.g. There are 45 rows in regcore_notice table and 5553 rows in regcore_document table for the year 202x.

Setup the local eregs API to run against the production eregs API

Without parsing regulations on local, run the eregs app by pointing to production eregs API.

• Change the API_BASE variable value in local_settings.py to point to production eregs API. It will show all 45 regulations for the year 202x.

    echo API_BASE = 'https://fec-prod-eregs.app.cloud.gov/regulations/api/'>> local_settings.py