Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate downstream projects to use the updated Python module naming convention #15

Closed
jacksondelametter opened this issue Apr 30, 2024 · 3 comments
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@jacksondelametter
Copy link
Contributor

jacksondelametter commented Apr 30, 2024

In aiSSEMBLE, the following Python modules were renamed to reflect aiSSEMBLE.

  • foundation-core-python → aissemble-foundation-core-python
  • foundation-model-training-api → aissemble-foundation-model-training-api
  • foundation-versioinig-service → aissemble-foundation-versioning-service
  • foundation-drift-detection-client → aissemble-drift-detection-client
  • foundation-enryption-policy-python → aissemble-encryption-policy-python
  • foundation-model-lineage → aissemble-foundation-model-lineage
  • foundation-data-lineage-python → aissemble-foundation-data-lineage-python
  • foundation-messaging-python-client → aissemble-foundation-message-python-client
  • foundation-pdp-client-python → aissemble-pdp-client-python
  • foundation-transform-core-python → aissemble-foundation-transfor-core-python
  • extensions-model-training-api-sagemaker → aissemble-extensions-model-training-api-sagemaker
  • extensions-data-delivery-spark-py → aissemble-extensions-data-delivery-spark-py
  • extensions-encryption-vault-python → aissemble-extensions-encryption-vault-python
  • extensions-transform-spark-python → aissemble-extensions-transform-spark-python
  • test-data-delivery-pyspark-model → aissemble-test-data-delivery-pyspark-model
  • test-data-delivery-pyspark-model-basic → aissemble-test-data-delivery-pyspark-model-basie
  • machine-learning-inference → aissemble-machine-learning-inference
  • machine-learning-training → aissemble-machine-learning-training
  • machine-learning-training-base → aissemble-machine-learning-training-base
  • machine-learning-sagemaker-training → aissemble-machine-learning-sagemaker-training

A migration feature needs to be added in order for current downstream projects to use the new Python modules when upgrading aiSSEMBLE.

@d-ryan-ashcraft d-ryan-ashcraft added the enhancement New feature or request label Apr 30, 2024
@d-ryan-ashcraft d-ryan-ashcraft added this to the 1.7.0 milestone Apr 30, 2024
aaron-gary added a commit that referenced this issue May 2, 2024
# This is the 1st commit message:

# This is a combination of 4 commits.
# This is the 1st commit message:

#2 Add Maven build workflow

# This is the commit message #2:

#2 update build workflow

# This is the commit message #3:

#2 add branch checkout to build workflow

# This is the commit message #4:

#2 Add on event to build workflow

# This is the commit message #2:

#2 Remove unused executions

# This is the commit message #3:

#2 Focus on build

# This is the commit message #4:

#2 Remove build execution where not needed

# This is the commit message #5:

#2 debug failing module

# This is the commit message #6:

#2 Remove unused target folder copy

# This is the commit message #7:

#2 Build spark and jenkins docker images

# This is the commit message #8:

#2 Retry full build

# This is the commit message #9:

#2 Omitting module

# This is the commit message #10:

#2 Fix for out of disk space

# This is the commit message #11:

#2 tagging docker images

# This is the commit message #12:

#2 Remove Temporarily remove Docker module

# This is the commit message #13:

#2 Build update

# This is the commit message #14:

#2 Build update

# This is the commit message #15:

#2 move chart dry-runs to IT profile

# This is the commit message #16:

#2 curl delta-hive assembly in docker build

# This is the commit message #17:

#2 cache m2 repo

# This is the commit message #18:

#2 prune docker build cache between images to save space

# This is the commit message #19:

#2 add maven build-cache to GH cache

# This is the commit message #20:

#2 run clean goal in build to clear docker cache

# This is the commit message #21:

#2 set maven caches to always save even if the build failed

# This is the commit message #22:

#2 adjust number of docker modules built

# This is the commit message #23:

#2 use the same cache for .m2 even if poms change

# This is the commit message #24:

#2 change from `save-always` flag to `if: always()` see actions/cache#1315

# This is the commit message #25:

#2 further reduce docker images being built

# This is the commit message #26:

#2 disable modules that depend on helm charts

# This is the commit message #27:

#2 use maven wrapper

# This is the commit message #28:

#2 restore modules to test build-cache

# This is the commit message #29:

#2 fix build of modules with intra-project chart dependencies

# This is the commit message #30:

#2 use explict .m2 repo cache so we can fall-back to older caches

# This is the commit message #31:

#2 save maven caches on build failure
aaron-gary added a commit that referenced this issue May 2, 2024
# This is the 1st commit message:

# This is a combination of 4 commits.
# This is the 1st commit message:

#2 Add Maven build workflow

# This is the commit message #2:

#2 update build workflow

# This is the commit message #3:

#2 add branch checkout to build workflow

# This is the commit message #4:

#2 Add on event to build workflow

# This is the commit message #2:

#2 Remove unused executions

# This is the commit message #3:

#2 Focus on build

# This is the commit message #4:

#2 Remove build execution where not needed

# This is the commit message #5:

#2 debug failing module

# This is the commit message #6:

#2 Remove unused target folder copy

# This is the commit message #7:

#2 Build spark and jenkins docker images

# This is the commit message #8:

#2 Retry full build

# This is the commit message #9:

#2 Omitting module

# This is the commit message #10:

#2 Fix for out of disk space

# This is the commit message #11:

#2 tagging docker images

# This is the commit message #12:

#2 Remove Temporarily remove Docker module

# This is the commit message #13:

#2 Build update

# This is the commit message #14:

#2 Build update

# This is the commit message #15:

#2 move chart dry-runs to IT profile

# This is the commit message #16:

#2 curl delta-hive assembly in docker build

# This is the commit message #17:

#2 cache m2 repo

# This is the commit message #18:

#2 prune docker build cache between images to save space

# This is the commit message #19:

#2 add maven build-cache to GH cache

# This is the commit message #20:

#2 run clean goal in build to clear docker cache

# This is the commit message #21:

#2 set maven caches to always save even if the build failed

# This is the commit message #22:

#2 adjust number of docker modules built

# This is the commit message #23:

#2 use the same cache for .m2 even if poms change

# This is the commit message #24:

#2 change from `save-always` flag to `if: always()` see actions/cache#1315

# This is the commit message #25:

#2 further reduce docker images being built

# This is the commit message #26:

#2 disable modules that depend on helm charts

# This is the commit message #27:

#2 use maven wrapper

# This is the commit message #28:

#2 restore modules to test build-cache

# This is the commit message #29:

#2 fix build of modules with intra-project chart dependencies

# This is the commit message #30:

#2 use explict .m2 repo cache so we can fall-back to older caches

# This is the commit message #31:

#2 save maven caches on build failure
@akanshamishra617
Copy link

Hi there,

I'm interested in contributing to the development of a migration feature for this module renaming.

I've noticed several core Python modules within aiSSEMBLE have been renamed to include the "aiassemble-" prefix. This is a great step for clarity and organization.

To ensure a smooth upgrade path for existing projects, a migration feature seems essential. I'd be happy to help develop this feature.

Here are some initial thoughts:

The migration feature could potentially map old module names to their corresponding new names during the upgrade process.
This would allow existing projects to continue functioning seamlessly after upgrading aiSSEMBLE.
I'm open to discussing different approaches and collaborating on a solution.

Would you be open to a pull request outlining a potential implementation? Additionally, are there any specific guidelines or preferences for contributing to this type of feature?

@d-ryan-ashcraft
Copy link
Collaborator

d-ryan-ashcraft commented May 3, 2024

@akanshamishra617

Appreciate your question and willingness to help!

This feature is almost complete - but you can definitely check it out the pull request and provide feedback there.

The general approach is consistent with how we have been working migrations on aissemble for a while now:

  • Whenever possible, migrations should seamlessly happen upon upgrade to the version of aissemble in which is becomes active with no manual intervention
  • If sensible, write a generic migration that can be configured to support the current need
  • Backup files automatically should captured automatically should the user need to roll back (though we expect projects to also be versioned)

We are using Baton to drive our migrations - it's a very simple open source project that facilitates lightweight, but structured migrations in manner that is very easy to integrate into the aissemble build process. You can find some other aissemble migrations here: https://github.com/boozallen/aissemble/tree/dev/foundation/foundation-upgrade (we've removed the old migrations from prior releases to reduce code bloat when migrating to this repository).

Thanks again!

@jacksondelametter
Copy link
Contributor Author

This feature was complete on May 3, should have been closed a few months ago!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants