Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend CRD: allow to define storageInitializerImage in the graph definition #2937

Merged
merged 6 commits into from
Feb 25, 2021
Merged

Extend CRD: allow to define storageInitializerImage in the graph definition #2937

merged 6 commits into from
Feb 25, 2021

Conversation

RafalSkolasinski
Copy link
Contributor

@RafalSkolasinski RafalSkolasinski commented Feb 8, 2021

What this PR does / why we need it:

This allows to define image used by the Storage Initializer as part of the PredictiveUnit definition - graph section in the CR e.g.

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: rclone-sklearn
spec:
  predictors:
  - name: default
    replicas: 1
    graph:
      name: classifier
      implementation: SKLEARN_SERVER
      modelUri: s3://sklearn/iris
      storageInitializerImage: gcr.io/kfserving/storage-initializer:v0.4.0
      envSecretRefName: seldon-init-container-secret

This allow to have elegant way to overwrite used storageInitializerImage on per-deployment basis.

Which issue(s) this PR fixes:

Fixes #2821

Special notes for your reviewer:

To be finished:

  • unit test
  • doc entry and example

Does this PR introduce a user-facing change?:

One can now define image used by `storageInitializer` init container by defining `storageInitializerImage` in the `graph` section of `SeldonDeployment` CR definition. 

@RafalSkolasinski
Copy link
Contributor Author

/test integration
/test notebooks

@RafalSkolasinski RafalSkolasinski changed the title initial attempt to customise storageInitializer per deployment Extend CRD: Add storageInitializerImage to the PredictiveUnit definition Feb 9, 2021
@RafalSkolasinski RafalSkolasinski changed the title Extend CRD: Add storageInitializerImage to the PredictiveUnit definition Extend CRD: allow to define storageInitializerImage in the graph definiion Feb 9, 2021
@RafalSkolasinski RafalSkolasinski changed the title Extend CRD: allow to define storageInitializerImage in the graph definiion Extend CRD: allow to define storageInitializerImage in the graph definition Feb 9, 2021
@seldondev
Copy link
Collaborator

Tue Feb 9 10:39:22 UTC 2021
The logs for [integration] [2] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2937/2.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2937 --build=2

@seldondev
Copy link
Collaborator

Tue Feb 9 10:39:23 UTC 2021
The logs for [notebooks] [1] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2937/1.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2937 --build=1

@seldondev seldondev added size/L and removed size/M labels Feb 9, 2021
@RafalSkolasinski RafalSkolasinski marked this pull request as ready for review February 9, 2021 18:13
@RafalSkolasinski
Copy link
Contributor Author

/test integration
/test notebooks

@RafalSkolasinski
Copy link
Contributor Author

This should be ready for review now.

@seldondev
Copy link
Collaborator

Tue Feb 9 18:15:15 UTC 2021
The logs for [integration] [4] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2937/4.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2937 --build=4

@seldondev
Copy link
Collaborator

Tue Feb 9 18:15:16 UTC 2021
The logs for [notebooks] [3] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2937/3.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2937 --build=3

@RafalSkolasinski
Copy link
Contributor Author

/test integration
/test notebooks

@seldondev
Copy link
Collaborator

Thu Feb 11 11:41:24 UTC 2021
The logs for [integration] [6] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2937/6.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2937 --build=6

@seldondev
Copy link
Collaborator

Thu Feb 11 11:41:28 UTC 2021
The logs for [notebooks] [5] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2937/5.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2937 --build=5

@ukclivecox
Copy link
Contributor

Need to rebase and rerun tests

IMAGE_VERSION := $(shell cat ../../version.txt)
IMAGE_NAME=seldonio/rclone-init-container-example
# IMAGE_VERSION := $(shell cat ../../version.txt)
IMAGE_VERSION=0.1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is image version not based on release. Is it assumed it is unlikely to change be effected by release. If so maybe remove the commented line?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only an example that actually has no dependency on the in-repo code. It just changes the entrypoint of the upstream image:

FROM rclone/rclone:latest
ENTRYPOINT ["rclone", "copy"]

When later we will add a proper rclone-based alternative to storage.py (not just an example of custom init container) then I guess it will make sense to version it with ../../version.txt.


manifests_all: manifests manifests_v1

install_all: install install_v1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

install all doesn't make sense to me - you either need to install pre 1.18 CRD or post 1.18 CRD

@@ -129,9 +133,10 @@ func (pi *PrePackedInitialiser) addTFServerContainer(mlDepSpec *machinelearningv
}

envSecretRefName := extractEnvSecretRefName(pu)
storageInitializerImage := extractStorageInitializerImage(pu)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need a separate function for this as it just returns the value from pu

@ukclivecox ukclivecox self-requested a review February 18, 2021 08:34
@seldondev
Copy link
Collaborator

failed to trigger Pull Request pipeline

  • failed to create agent
  • failed to calculate in repo config
  • failed to load trigger config for repository SeldonIO/seldon-core for ref e9cde00
  • failed to find any lighthouse configuration files in repo SeldonIO/seldon-core at sha e9cde00
  • failed to process repo SeldonIO/seldon-core refref e9cde00
  • failed to list files in directory /var/tmp/gitrepo598506571/.lighthouse
  • open /var/tmp/gitrepo598506571/.lighthouse
  • no such file or directory

Copy link
Contributor

@ukclivecox ukclivecox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few small changes

@RafalSkolasinski
Copy link
Contributor Author

/test integration
/test notebooks

@RafalSkolasinski
Copy link
Contributor Author

/test notebooks

@ukclivecox
Copy link
Contributor

/approve

@seldondev
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cliveseldon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cannot overwrite initContainers image: reconcile error
3 participants