Releases: climatepolicyradar/navigator-data-ingest
v2.0.5-beta
Removing the glob statements to find files for an id and replacing this with well know document paths created using f-strings. This is a less dynamic solution but cloudpathlib takes a very long time to search the entire s3 prefix for the file.
v2.0.4-beta
Updating the fields of the update type from csv_value and db_value to db_value and s3_value.
v2.0.3-beta
Update the ingest stage to trust the db_state.json provided value and overwrite any data in s3 even if it is not as expected. The data mismatch is instead logged.
v2.0.2-beta
Updating of the parse action to archive all instances of the document in the pipeline as this is the required functionality now.
v2.0.1-beta
- Refactoring code
- Updating the type of the Document that is passed from the backend in the db_state.json file to no longer contain document_status
v2.0.0-beta
Updating the ingest stage to read in document updates and deletes. This is a document that contains the information on new documents and the updates to be made to existing documents. The ingest stage then contains functionality to facilitate updates to the updated documents.
v1.0.2-beta
Hardcoding the s3 path for the pdf document upload to s3 to use the navigator/ prefix.
v1.0.1-beta
Updating of the logs for document upload to include the import ID.
v1.0.0-beta
First release for new infra