Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SOF-8 Refactor for ingest #114

Merged
merged 40 commits into from
May 22, 2023
Merged

SOF-8 Refactor for ingest #114

merged 40 commits into from
May 22, 2023

Conversation

diversemix
Copy link
Contributor

@diversemix diversemix commented May 19, 2023 •
edited
Loading

Description

Get ingest working

Type of change

Please select the option(s) below that are most relevant:

  • Bug fix
  • New feature
  • Breaking change

How Has This Been Tested?

Please describe the tests that you added to verify your changes.

Reviewer Checklist

  • The PR represents a single feature (small driveby fixes are also ok)
  • The PR includes tests that are sufficient for the level of risk
  • The code is sufficiently commented, particularly in hard-to-understand areas
  • Any required documentation updates have been made
  • Any TODOs added are captured in future tickets
  • No FIXMEs remain

diversemix and others added 30 commits May 15, 2023 16:13
@diversemix
begin re-factor of how taxonomies are populated
8c5fced
@diversemix
refactor
f033042
@diversemix
add type to arg
146c739
@diversemix
Add a little more documentation for taxonomy
2faca37
@diversemix
More refactoring and add tests
0c4fa07
@diversemix
finish tests
de8bb08
@diversemix
Merge branch 'main' into SOF-8/ingest-UNF3C
e2c727d
@diversemix
First re-factor to accomodate UNFCCC
febd157
@diversemix
Merge branch 'main' into SOF-8/ingest-UNF3C
218eb33
@diversemix
get tests working (not)
76baa6b
@diversemix
lunch time commit
7b9bb94
@diversemix
fix datetime issue
0c672d3
@diversemix
more re-factoring
21a2b59
@diversemix
end of day commit
c6b6cd9
@diversemix
get validation working
3940d7b
@diversemix
start implementing the collections csv
c874baa
@diversemix
Validate the Collections CSV
63d5af4
@diversemix
add more tests
dfe56b0
@diversemix
add database migration
ab7ce36
@diversemix
Update app/core/ingestion/processor.py
d699923 
Co-authored-by: Joel Wright <[email protected]>
@diversemix
remove law-policy from the routes
c4710d9
@diversemix
create sub-modules
a32ad26
@diversemix
fully validate collections
af5be24
@diversemix
update field names for collection csv
ffc4562
@diversemix
Add new CSV columns
af6440d
@diversemix
remove collection_id as no longer used
1211a5b
@diversemix
write test but comment out - pivoting
75623ac
@diversemix
remove submission type from taxonomy and add to document type
7bace38
@diversemix
fix failing populate_taxonomy test
2196d1c
@diversemix
more ingest work
6ecf87a
@linear
Copy link

linear bot commented May 19, 2023

SOF-8 New ingest endpoint for GST

Similar to that for CCLW - but this will read the new input CSV schema and create the necessary data and structure in the database

@diversemix diversemix marked this pull request as ready for review May 19, 2023 13:50
@diversemix diversemix added this pull request to the merge queue May 22, 2023
Merged via the queue into main with commit a4c363c May 22, 2023
@diversemix diversemix deleted the SOF-8/part-2-ingest branch May 22, 2023 09:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@joel-wright joel-wright joel-wright approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@diversemix @joel-wright