[feature] Allow users to skip docker build in kfp components build

[ghost]

Feature Area

/area sdk

What feature would you like to see?

Building containerised components using the Python DSL, and packaging them using kfp components build is a really nice user experience - you get the benefits of the Python DSL without some of the drawbacks of "lightweight" components (e.g. having to install dependencies at runtime).

However, there are some scenarios where building container images using Docker is not an option, for example:

• Some organisations do not allow Docker to be installed on Data Scientists' laptops (very common for enterprise users)
• Some CI/CD platforms do not support docker-in-docker, and use other mechanisms (such as kaniko) for building container images

This feature would allow the user to skip the actual container image build, but still perform the other functions in kfp components build (generation of the different files). The user would then be responsible for building the container images using their preferred mechanism (e.g. Cloud Build, Kaniko)

What is the use case or pain point?

Currently not all users are able to use the kfp components build command as they cannot run Docker (locally or in CI/CD)

Is there a workaround currently?

Current workarounds:

• Roll your own containerised components without the convenience of the kfp components build command
• I suppose you could mock the docker library as is done in the tests? But this is quite hacky!

---

Love this idea? Give it a 👍.

[chensun]

The feature request makes sense to me. In terms of implementation, have you considered adding support for different engines?

pipelines/sdk/python/kfp/deprecated/cli/components.py

Lines 72 to 76 in 1f5d8ff

	class _Engine(str, enum.Enum):
	"""Supported container build engines."""
	DOCKER = 'docker'
	KANIKO = 'kaniko'
	CLOUD_BUILD = 'cloudbuild'

[ghost]

Hi Chen! Thanks for your quick response!

I took a brief look at other container build engines. It looks like there is no official Python package for kaniko (although there is the unofficial PyKaniko). Cloud Build could be supported using the Cloud Build Python SDK, but there are a lot of potential build options to account for! There are also a number of other build engines that users might prefer for one reason or another e.g. buildah.

I think an ideal solution would be to add support for more container build engines (tbd which ones we support in the SDK), and also to allow the user to skip the container build step and build the containers themselves (e.g. in a subsequent step of a CI/CD pipeline or bash script)

What do you think?

[chensun]

Makes sense to me.
/cc @connor-mccarthy for a second opinion.

[connor-mccarthy]

Agreed. Even with other build engine options, I think it makes sense to enable the "no engine" option also.

[ghost]

Thanks both! In terms of next steps, I propose

• Merge in feat(sdk): add --build-image option to 'kfp components build' to allow users to skip docker build. Fixes #8382 for 1.8 #8383 - thanks!
• Add the "no engine" option to v2 in master - I will work on a PR for this
• Implement a Cloud Build option

Looking more at kaniko, I'm not sure how this would work as kaniko is designed to run in its own container.

Cloud Build seems like a useful option to have, particularly for users who can't run container locally, so I'm happy to work on that. As I mentioned, there are a number of options that would need to be passed to Cloud Build e.g. project ID, region, staging bucket etc. How would you like these to be exposed to the user? My initial thought is to add additional command-line flags --cloudbuild-project, --cloudbuild-region, --cloudbuild-staging-bucket etc. Does that sound reasonable, or do you have another preferred approach?

[connor-mccarthy]

Thanks, @browningjp-datatonic!

Add the "no engine" option to v2 in master - I will work on a PR for this

We probably ought to use the same --no-build-image to conform with #8383 and ease migration from KFP SDK v1 to v2.

How would you like these to be exposed to the user?

I think adding engine-specific flags is a non-option, unfortunately, as parameterizing kfp components build for each engine will quickly bloat the CLI's API. Are there other options for exposing these?

[ghost]

We probably ought to use the same --no-build-image to conform with #8383 and ease migration from KFP SDK v1 to v2.

Yes, I will make sure they match

I think adding engine-specific flags is a non-option, unfortunately, as parameterizing kfp components build for each engine will quickly bloat the CLI's API. Are there other options for exposing these?

I am struggling a bit for good options tbh! These are the ones that come to mind:

1. Engine-specific flags - will quickly bloat the API as you say
2. Some kind of config file - this is then another spec to design, maintain, and implement for each engine
3. Environment variables - if a particular engine accepts options as env variables already, great (nothing to do), otherwise we have a naming convention to work out, maintain, implement...
4. A generic --build-options flag containing comma-separated key-value pairs? E.g. --build-options=project=my-gcp-project,region=europe-west2,staging_bucket=gs://my-gcs-bucket. I think I like this option the most as it doesn't bloat the API (directly at least) and gives the flexibility for whatever options an engine might need. The drawback I can see is that it is fiddly to document well.
5. (Something else I've not thought of...)

What do you think?

[connor-mccarthy]

@browningjp-datatonic

I don't have any other options in mind.

Is cloudbuild the engine you are using? I'm curious if there are many user requests for this engine. If not, perhaps we don't need to rush to implement this engine.

[ghost]

Yes, we would certainly use Cloud Build. All our customers are on Google Cloud anyway, and most of them are enterprise customers that can't use Docker locally.

That said, it's very easy to just run kfp components build --no-build-image . && gcloud builds submit ... and wrap that in a script or Makefile, so I agree it's not an urgent need.

I notice that in the master branch, the docstring for the --engine flag says it is deprecated. If we are considering implementing more engines, should we remove this before the next stable release, so that we don't end up having to un-deprecate the flag?

[connor-mccarthy]

I think kfp components build --no-build-image . && gcloud builds submit ... is a good approach.

We probably don't want to be in the business of maintaining integrations with many different build providers and the KFP SDK provides only a very thin wrapper around the commands you shared.

For example:
kfp components build --no-build-image . && gcloud builds submit --cloudbuild-project ... becomes
kfp components build --no-build-image --cloudbuild-project ... .. The diff is gcloud builds submit. There is not much typing saved and it comes at the cost of (1) transparency about what is going on and (2) debuggability for the user via an added layer of indirection.

Thank you for bringing this up @browningjp-datatonic! I will review #8387.

[ghost]

I agree - don't think it is worth the maintenance burden to support additional engines. I'm happy that this issue can be closed/resolved by #8383 and #8387

[ghost]

BTW - what is the usual cadence for new releases for 1.8? We have a current customer use case that this would be extremely useful for!

[connor-mccarthy]

@browningjp-datatonic, we don't have a scheduled cadence for 1.8 releases. We typically wait until features accumulate to minimize having an abundance of patch releases with only a few number of changes.

Until the next release, you can pip install from GitHub HEAD. Does that unblock you for now?

[ghost]

@connor-mccarthy no problem. The customer is not able to install Python packages from GitHub directly (everything goes through a private PyPi mirror), but we can make a start on development in a less restrictive environment.

Thanks for all your help!