Skip to content

Commit

Permalink
Add official support for python 3.9 (#687)
Browse files Browse the repository at this point in the history
* Run 3.9 in CI

Signed-off-by: Eduardo Apolinario <[email protected]>

* Also run 3.9 in plugins tests.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix tests

Signed-off-by: Eduardo Apolinario <[email protected]>

* Account for the different exception type raised in case of wrong types

Signed-off-by: Eduardo Apolinario <[email protected]>

* Exclude spark2

Signed-off-by: Eduardo Apolinario <[email protected]>

* Failed to load json_data to dataclass (#684)

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>

* Remove sync singledispatch, add option for top-level only sync (#681)

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>

* Failed to transform path string to Literal (#689)

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>

* Don't node sync on remote wait (#690)

* no node sync

Signed-off-by: Yee Hing Tong <[email protected]>

* add param to wait

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix plugin regressions (#688)

* fix pandera regression

Signed-off-by: Niels Bantilan <[email protected]>

* install plugin with pip

Signed-off-by: Niels Bantilan <[email protected]>

* fix pandera plugin tests

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* add spark flytekit plugin to papermill test_requires

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* add sqlalchemy to great expectations plugin

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* plugins plugins plugins!

Signed-off-by: Niels Bantilan <[email protected]>

* lint

Signed-off-by: Niels Bantilan <[email protected]>

* Exclude does not understand lists

Signed-off-by: Eduardo Apolinario <[email protected]>

* Invert python version check

Signed-off-by: Eduardo Apolinario <[email protected]>

* Invert python version check for real this time

Signed-off-by: Eduardo Apolinario <[email protected]>

* Enable 3.10 just for kicks

Signed-off-by: Eduardo Apolinario <[email protected]>

* Add quotes around python versions

Yes, this is needed, please see https://dev.to/hugovk/the-python-3-1-problem-85g.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Revert "Add quotes around python versions"

This reverts commit 4d619d5.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Revert "Enable 3.10 just for kicks"

This reverts commit bd6d694.

Signed-off-by: Eduardo Apolinario <[email protected]>

* wip - restricted types

Signed-off-by: Eduardo Apolinario <[email protected]>

* wip - restricted types

Signed-off-by: Eduardo Apolinario <[email protected]>

* Comment use of restricted types in get_transformer

Signed-off-by: Eduardo Apolinario <[email protected]>

* Publish 3.9 image

Signed-off-by: Eduardo Apolinario <[email protected]>

* Add python 3.9 to the list of supported languages

Signed-off-by: Eduardo Apolinario <[email protected]>

* Comment RestrictedTypeTransformer and add one test case.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Review feedback

Signed-off-by: Eduardo Apolinario <[email protected]>

* Comment RestrictedTypeTransformer

Signed-off-by: Eduardo Apolinario <[email protected]>

* Handle the TypeError

Signed-off-by: Eduardo Apolinario <[email protected]>

* Remove breakpoint

Signed-off-by: Eduardo Apolinario <[email protected]>

Co-authored-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
Co-authored-by: Niels Bantilan <[email protected]>
  • Loading branch information
5 people authored Oct 19, 2021
1 parent eeb66b3 commit ae15c4e
Show file tree
Hide file tree
Showing 6 changed files with 81 additions and 14 deletions.
6 changes: 4 additions & 2 deletions .github/workflows/pythonbuild.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,13 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.7, 3.8]
python-version: [3.7, 3.8, 3.9]
spark-version-suffix: ["", "-spark2"]
exclude:
- python-version: 3.8
spark-version-suffix: "-spark2"
- python-version: 3.9
spark-version-suffix: "-spark2"
steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
Expand Down Expand Up @@ -52,7 +54,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.8]
python-version: [3.8, 3.9]
plugin-names:
- flytekit-aws-athena
- flytekit-aws-sagemaker
Expand Down
14 changes: 14 additions & 0 deletions .github/workflows/pythonpublish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,3 +80,17 @@ jobs:
build_extra_args: "--compress=true --build-arg=VERSION=${{ steps.bump.outputs.version }} --build-arg=DOCKER_IMAGE=ghcr.io/flyteorg/flytekit:py38-${{ steps.bump.outputs.version }}"
context: .
dockerfile: Dockerfile.py38
- name: Build & Push Flytekit Python3.9 Docker Image to Github Registry
uses: whoan/docker-build-with-cache-action@v5
with:
# https://docs.github.com/en/packages/learn-github-packages/publishing-a-package
username: "${{ secrets.FLYTE_BOT_USERNAME }}"
password: "${{ secrets.FLYTE_BOT_PAT }}"
image_name: ${{ github.repository_owner }}/flytekit
image_tag: py39-latest,py39-${{ github.sha }},py39-${{ steps.bump.outputs.version }}
push_git_tag: true
push_image_and_stages: true
registry: ghcr.io
build_extra_args: "--compress=true --build-arg=VERSION=${{ steps.bump.outputs.version }} --build-arg=DOCKER_IMAGE=ghcr.io/flyteorg/flytekit:py39-${{ steps.bump.outputs.version }}"
context: .
dockerfile: Dockerfile.py39
16 changes: 16 additions & 0 deletions Dockerfile.py39
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
FROM python:3.9-slim-buster

MAINTAINER Flyte Team <[email protected]>
LABEL org.opencontainers.image.source https://github.com/flyteorg/flytekit

RUN pip install awscli
RUN pip install gsutil

ARG VERSION
ARG DOCKER_IMAGE

RUN pip install -U flytekit==$VERSION

WORKDIR /app

ENV FLYTE_INTERNAL_IMAGE "$DOCKER_IMAGE"
48 changes: 37 additions & 11 deletions flytekit/core/type_engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import mimetypes
import typing
from abc import ABC, abstractmethod
from typing import Optional, Type, cast
from typing import NamedTuple, Optional, Type, cast

from dataclasses_json import DataClassJsonMixin, dataclass_json
from google.protobuf import json_format as _json_format
Expand Down Expand Up @@ -149,9 +149,10 @@ class RestrictedTypeError(Exception):
pass


class RestrictedType(TypeTransformer[T], ABC):
class RestrictedTypeTransformer(TypeTransformer[T], ABC):
"""
A Simple implementation of a type transformer that uses simple lambdas to transform and reduces boilerplate
Types registered with the RestrictedTypeTransformer are not allowed to be converted to and from literals. In other words,
Restricted types are not allowed to be used as inputs or outputs of tasks and workflows.
"""

def __init__(self, name: str, t: Type[T]):
Expand All @@ -160,6 +161,12 @@ def __init__(self, name: str, t: Type[T]):
def get_literal_type(self, t: Type[T] = None) -> LiteralType:
raise RestrictedTypeError(f"Transformer for type {self.python_type} is restricted currently")

def to_literal(self, ctx: FlyteContext, python_val: T, python_type: Type[T], expected: LiteralType) -> Literal:
raise RestrictedTypeError(f"Transformer for type {self.python_type} is restricted currently")

def to_python_value(self, ctx: FlyteContext, lv: Literal, expected_python_type: Type[T]) -> T:
raise RestrictedTypeError(f"Transformer for type {self.python_type} is restricted currently")


class DataclassTransformer(TypeTransformer[object]):
"""
Expand Down Expand Up @@ -339,10 +346,15 @@ class TypeEngine(typing.Generic[T]):
"""

_REGISTRY: typing.Dict[type, TypeTransformer[T]] = {}
_RESTRICTED_TYPES: typing.List[type] = []
_DATACLASS_TRANSFORMER: TypeTransformer = DataclassTransformer()

@classmethod
def register(cls, transformer: TypeTransformer, additional_types: Optional[typing.List[Type]] = None):
def register(
cls,
transformer: TypeTransformer,
additional_types: Optional[typing.List[Type]] = None,
):
"""
This should be used for all types that respond with the right type annotation when you use type(...) function
"""
Expand All @@ -356,6 +368,15 @@ def register(cls, transformer: TypeTransformer, additional_types: Optional[typin
)
cls._REGISTRY[t] = transformer

@classmethod
def register_restricted_type(
cls,
name: str,
type: Type,
):
cls._RESTRICTED_TYPES.append(type)
cls.register(RestrictedTypeTransformer(name, type))

@classmethod
def get_transformer(cls, python_type: Type) -> TypeTransformer[T]:
"""
Expand Down Expand Up @@ -399,10 +420,15 @@ def get_transformer(cls, python_type: Type) -> TypeTransformer[T]:
for base_type in cls._REGISTRY.keys():
if base_type is None:
continue # None is actually one of the keys, but isinstance/issubclass doesn't work on it
if isinstance(python_type, base_type) or (
inspect.isclass(python_type) and issubclass(python_type, base_type)
):
return cls._REGISTRY[base_type]
try:
if isinstance(python_type, base_type) or (
inspect.isclass(python_type) and issubclass(python_type, base_type)
):
return cls._REGISTRY[base_type]
except TypeError:
# As of python 3.9, calls to isinstance raise a TypeError if the base type is not a valid type, which
# is the case for one of the restricted types, namely NamedTuple.
logger.debug(f"Invalid base type {base_type} in call to isinstance", exc_info=True)
raise ValueError(f"Type {python_type} not supported currently in Flytekit. Please register a new transformer")

@classmethod
Expand Down Expand Up @@ -896,9 +922,9 @@ def _register_default_type_transformers():
# that the return signature of a task can be a NamedTuple that contains another NamedTuple inside it.
# Also, it's not entirely true that Flyte IDL doesn't support tuples. We can always fake them as structs, but we'll
# hold off on doing that for now, as we may amend the IDL formally to support tuples.
TypeEngine.register(RestrictedType("non typed tuple", tuple))
TypeEngine.register(RestrictedType("non typed tuple", typing.Tuple))
TypeEngine.register(RestrictedType("named tuple", typing.NamedTuple))
TypeEngine.register_restricted_type("non typed tuple", tuple)
TypeEngine.register_restricted_type("non typed tuple", typing.Tuple)
TypeEngine.register_restricted_type("named tuple", NamedTuple)


_register_default_type_transformers()
1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Topic :: Scientific/Engineering",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
"Topic :: Software Development",
Expand Down
10 changes: 9 additions & 1 deletion tests/flytekit/unit/core/test_type_hints.py
Original file line number Diff line number Diff line change
Expand Up @@ -685,7 +685,15 @@ def my_wf(a: int, b: str) -> (int, str):
assert context_manager.FlyteContextManager.size() == 1


def test_cant_use_normal_tuples():
def test_cant_use_normal_tuples_as_input():
with pytest.raises(RestrictedTypeError):

@task
def t1(a: tuple) -> str:
return a[0]


def test_cant_use_normal_tuples_as_output():
with pytest.raises(RestrictedTypeError):

@task
Expand Down

0 comments on commit ae15c4e

Please sign in to comment.