SOF-8 First re-factor to accomodate UNFCCC (#113)

* begin re-factor of how taxonomies are populated

* refactor

* add type to arg

* Add a little more documentation for taxonomy

* More refactoring and add tests

* finish tests

* First re-factor to accomodate UNFCCC

* get tests working (not)

* lunch time commit

* fix datetime issue

* more re-factoring

* end of day commit

* get validation working

* start implementing the collections csv

* Validate the Collections CSV

* add more tests

* add database migration

* Update app/core/ingestion/processor.py

Co-authored-by: Joel Wright <[email protected]>

* remove law-policy from the routes

* create sub-modules

* fully validate collections

* update field names for collection csv

* Add new CSV columns

* remove collection_id as no longer used

* write test but comment out - pivoting

* remove submission type from taxonomy and add to document type

* fix failing populate_taxonomy test

* more ingest work

* more tests ready for writing ingest code

* Allow pre-existing collection if they match

* add download_url

---------

Co-authored-by: Joel Wright <[email protected]>

alembic/versions/0016_add_unfccc.py

@@ -0,0 +1,30 @@
+"""remove deprecated db models
+
+Revision ID: 0016
+Revises: 0015
+Create Date: Manually see commit
+
+"""
+from alembic import op
+from alembic.op import execute
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+# revision identifiers, used by Alembic.
+revision = '0016'
+down_revision = '0015'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    execute("ALTER TYPE familycategory ADD VALUE 'UNFCCC'")
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    # Downgrading is not that simple see: https://stackoverflow.com/a/56777227
+    raise ValueError("We really don't want to be here")
+    # ### end Alembic commands ###