Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement ETL process for migrating Schemes & Concepts from RDM tables to Lingo models #6

Merged
merged 53 commits into from
May 21, 2024
Merged

Implement ETL process for migrating Schemes & Concepts from RDM tables to Lingo models #6

merged 53 commits into from
May 21, 2024

Conversation

johnatawnclementawn
Copy link
Member

@johnatawnclementawn johnatawnclementawn commented Feb 7, 2024 •
edited
Loading

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Description of Change

Uses the Arches ETL manager to extract & transform schemes and concepts from an existing Arches Reference Data Manager schema into the Arches Lingo Scheme and Concept Models.

Issues Solved

Resolves #2

Checklist

  • Unit tests pass locally with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Ticket Background

  • Sponsored by:
  • Found by: @
  • Tested by: @
  • Designed by: @

Further comments

@johnatawnclementawn
Create RDM Migrator ETL module
81bf21d
@johnatawnclementawn
Stub out rdm migrator backend #2
f2fd6e8
@johnatawnclementawn
Stub out RDM migrator front end #2
6595cd1
@johnatawnclementawn
Move file to proper location #2
cd49d28
@johnatawnclementawn
Add migrate rdm task
e0f159c
@johnatawnclementawn
Gather graphs & init rdm migration on click submit #2
5317703
@johnatawnclementawn
Update urls.py
0fa7faf
@johnatawnclementawn
Add further dependency and show/hide etl manager
38fc806
@johnatawnclementawn
Add button to kick off migration
da587c8
@johnatawnclementawn
Undo changes that caused malformed requests
bd95c4f
@johnatawnclementawn
Remove unused imports
d91cf4b
@johnatawnclementawn
Place module in correct path #2
0ee7711
@johnatawnclementawn
Make RDM migrator child class of BaseImport and cleanup unused imports …
08d6dd6 
…#2
@johnatawnclementawn
Make RDM migrator importer, not editor
00f9e1f
@johnatawnclementawn
nit #2
0e3fc53
@johnatawnclementawn
Working on constructing tile data #2
deda1a3
@johnatawnclementawn
Functional tile value generator #2
03c4e7e
@johnatawnclementawn
Loading data into staging table #2
4b5994a
@johnatawnclementawn
nit #2
b6e4acf
@johnatawnclementawn
nit #2
407d211
@johnatawnclementawn
Improve control flow and pass existing cursor #2
cb3e40d
@johnatawnclementawn
Improve control flow #2
ed2586a
@johnatawnclementawn
Functional RDM etl creating new resources #2
58fb79d
@johnatawnclementawn
nit #2
3165091
@johnatawnclementawn
nit #2
4c2b383
@johnatawnclementawn
Functional ETL module for migrating RDM #2
9e8feee
@johnatawnclementawn
Make rdm migrator reversible #2
c6af024
@johnatawnclementawn
Stub out using bulk_create #2
ec90031
@johnatawnclementawn
django will serialize json going into the db #2
be7745b
jacobtylerwalls
jacobtylerwalls reviewed Feb 8, 2024
View reviewed changes
Copy link
Member

@jacobtylerwalls jacobtylerwalls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice progress!

arches_rdm/media/js/views/components/etl_modules/migrate-rdm.js Outdated Show resolved Hide resolved
arches_rdm/media/js/views/components/etl_modules/migrate-rdm.js Outdated Show resolved Hide resolved
arches_rdm/templates/views/components/etl_modules/migrate-rdm.htm Outdated Show resolved Hide resolved
arches_rdm/tasks.py Outdated Show resolved Hide resolved
arches_rdm/etl_modules/migrate_rdm.py Outdated Show resolved Hide resolved
arches_rdm/templates/views/components/etl_modules/migrate-rdm.htm Outdated Show resolved Hide resolved
@jacobtylerwalls jacobtylerwalls mentioned this pull request Mar 1, 2024
Merged
@johnatawnclementawn johnatawnclementawn marked this pull request as ready for review March 5, 2024 22:18
@johnatawnclementawn johnatawnclementawn changed the title Implement ETL process for migrating RDM to ADM Implement ETL process for migrating RDM to Lingo models Mar 5, 2024
@johnatawnclementawn johnatawnclementawn requested review from robgaston and removed request for robgaston March 5, 2024 22:26
@jacobtylerwalls jacobtylerwalls mentioned this pull request Mar 12, 2024
Closed
@johnatawnclementawn johnatawnclementawn changed the title Implement ETL process for migrating RDM to Lingo models Implement ETL process for migrating Schemes & Concepts from RDM tables to Lingo models Apr 24, 2024
@robgaston robgaston merged commit d6e0a39 into main May 21, 2024
@johnatawnclementawn johnatawnclementawn deleted the 2_RDM_Migration_ETL branch May 21, 2024 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jacobtylerwalls jacobtylerwalls jacobtylerwalls left review comments

@robgaston robgaston robgaston approved these changes

Assignees

@robgaston robgaston

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Script the ETL of data from the existing RDM tables into the new resource models
3 participants
@johnatawnclementawn @robgaston @jacobtylerwalls