Skip to content

Commit

Permalink
feat(sdk): Enable containerizing v2 Python components (#6417)
Browse files Browse the repository at this point in the history
* Refactor and move all v2 related code to under the v2 namespace.

Most of the changes are around imports and restructuring of the
codebase. While it looks like a lot of code was added, most of the code
already existed and was simply moved or copied over to v2. The only
exceptions are:
- under kfp/v2/components/component_factory.py: some helper functions
  were copied with simplification from _python_op.py
- we no longer strip the `_path` suffix in v2 components.

Note: there is still some duplication of code (particularly between
component_factory.py and _python_op.py), but it's ok for now since we
intend to replace some of this with v2 ComponentSpec + BaseComponent.

* Update setup.py.

* update tests.

* revert accidental change of gcpc

* Fix component entrypoint.

* Update goldens.

* fix tests.

* fix merge conflict.

* revert gcpc change.

* fix tests.

* fix tests.

* Add type aliases for moved files.

* merge and update goldens.

* Add a CLI command to help containerize and build v2 components.

* Flesh out the CLI for Docker.

* update comments.

* Support multiple component files in the same directory.

* address pr comments.

* Let user specify just the directory containing components.

The CLI will search for components in all Python files by default. It
can also search for a specific filepattern (as supported by
pathlib.Path objects).

Also add a bunch of tests.

* Clean up.

* fix merge error.

* Fix docker import error for tests.

* Update release notes.

* address PR comments.
  • Loading branch information
neuromage authored Oct 12, 2021
1 parent 991473d commit fe66377
Show file tree
Hide file tree
Showing 13 changed files with 1,344 additions and 165 deletions.
4 changes: 4 additions & 0 deletions sdk/RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
## Major Features and Improvements

* Add v2 placeholder variables [\#6693](https://github.com/kubeflow/pipelines/pull/6693)
* Add a new command in KFP's CLI, `components`, that enables users to manage and build
v2 components in a container with Docker [\#6417](https://github.com/kubeflow/pipelines/pull/6417)


## Breaking Changes

Expand All @@ -22,6 +25,7 @@
* Depends on `typing-extensions>=3.7.4,<4; python_version<"3.9"` [\#6683](https://github.com/kubeflow/pipelines/pull/6683)
* Depends on `click>=7.1.2,<9` [\#6691](https://github.com/kubeflow/pipelines/pull/6691)
* Depends on `cloudpickle>=2.0.0,<3` [\#6703](https://github.com/kubeflow/pipelines/pull/6703)
* Depends on `typer>=0.3.2,<1.0` [\#6417](https://github.com/kubeflow/pipelines/pull/6417)

## Documentation Updates

Expand Down
7 changes: 6 additions & 1 deletion sdk/python/kfp/cli/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.

import click
import logging
import sys

import click
import typer

from kfp._client import Client
from kfp.cli.run import run
from kfp.cli.pipeline import pipeline
from kfp.cli.diagnose_me_cli import diagnose_me
from kfp.cli.experiment import experiment
from kfp.cli.output import OutputFormat
from kfp.cli import components


@click.group()
Expand Down Expand Up @@ -67,6 +71,7 @@ def main():
cli.add_command(pipeline)
cli.add_command(diagnose_me, 'diagnose_me')
cli.add_command(experiment)
cli.add_command(typer.main.get_command(components.app))
try:
cli(obj={}, auto_envvar_prefix='KFP')
except Exception as e:
Expand Down
Loading

0 comments on commit fe66377

Please sign in to comment.