Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix sagemaker orchestrator and step operator env vars and other minor bugs #1993

Merged
merged 9 commits into from
Oct 30, 2023

Conversation

stefannica
Copy link
Contributor

@stefannica stefannica commented Oct 26, 2023

Describe changes

This PR fixes the following issues:

  • Sagemaker has a limit on the size of the environment variable values passed to processing steps (256) and estimator steps (512). We exceed the 256 limit with our API Tokens now. Fixed by splitting large env var values into multiple chunks and reconstructing them on the other side
  • the AWS container registry still runs into errors if the local AWS credentials don't have access to check that the repository exists
  • zenml orchestrator flavor describe sagemaker and zenml step-operator flavor describe sagemaker fail because the configuration includes attributes that have a Union type and that is not currently supported by the CLI

Pre-requisites

Please ensure you have done the following:

  • I have read the CONTRIBUTING.md document.
  • If my change requires a change to docs, I have updated the documentation accordingly.
  • If I have added an integration, I have updated the integrations table and the corresponding website section.
  • I have added tests to cover my changes.
  • I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Other (add details above)

@github-actions github-actions bot added internal To filter out internal PRs and issues bug Something isn't working labels Oct 26, 2023
Copy link
Contributor

@schustmi schustmi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this environment variable change also be necessary for the Sagemaker Step Operator?

@strickvl
Copy link
Contributor

Would this environment variable change also be necessary for the Sagemaker Step Operator?

I imagine it does. Step Operator use Estimator steps and Orchestrator uses Processor steps, but they both allow env variables to be passed in. And from what I understood, the limitation wasn't clearly documented on AWS so we'll just have to try and find out.

@strickvl
Copy link
Contributor

This logic looks good, though would love to have some test(s) added since this added functionality is fairly fiddly.

@stefannica stefannica changed the title Fix sagemaker orchestrator env vars and other minor bugs Fix sagemaker orchestrator and step operator env vars and other minor bugs Oct 27, 2023
@stefannica stefannica merged commit eb32d88 into develop Oct 30, 2023
30 checks passed
@stefannica stefannica deleted the bug/fix-sagemaker-orchestrator branch October 30, 2023 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working internal To filter out internal PRs and issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants