TypeTransformer for Keras

[ryankarlos]

Signed-off-by: Ryan Nazareth [email protected]

TL;DR

Adds a typetransformer to support keras.Model and keras.Sequential as native types

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

• __init__.py file handles case where the user doesn't have keras installed.
• TypeTransformer implemented for the following classes: Sequential: https://keras.io/api/models/sequential/
• added tests in https://github.com/flyteorg/flytekit/tree/master/tests/flytekit/unit/extras/keras

Tracking Issue

flyteorg/flyte#2759

Follow-up issue

N/A

[ryankarlos]

see my comment in #1240 (comment) for reasons for pinning tf versions.

[ryankarlos]

Ignore this - ive pinned grpcio-status<1.49.0 instead based on suggestion from @pingsutw in another PR, which fixed it !

[pingsutw]

There is an error in test_to_python_value_and_literal

    def test_to_python_value_and_literal(transformer, python_type, format, python_val):
        ctx = context_manager.FlyteContext.current_context()
        tf = transformer
        lt = tf.get_literal_type(python_type)
        lv = tf.to_literal(ctx, python_val, type(python_val), lt)  # type: ignore
        assert lv.scalar.blob.metadata == BlobMetadata(
            type=BlobType(
                format=format,
                dimensionality=BlobType.BlobDimensionality.SINGLE,
            )
        )
        assert lv.scalar.blob.uri is not None
    
        output = tf.to_python_value(ctx, lv, python_type)
        if isinstance(python_val, keras.Sequential):
            for p1, p2 in zip(output.weights, python_val.weights):
                np.testing.assert_array_equal(p1.numpy(), p2.numpy())
            assert True
        else:
>           assert isinstance(output, dict)
E           assert False
E            +  where False = isinstance(<keras.engine.functional.Functional object at 0x000001AE37320108>, dict)

[ryankarlos]

There is an error in test_to_python_value_and_literal

    def test_to_python_value_and_literal(transformer, python_type, format, python_val):
        ctx = context_manager.FlyteContext.current_context()
        tf = transformer
        lt = tf.get_literal_type(python_type)
        lv = tf.to_literal(ctx, python_val, type(python_val), lt)  # type: ignore
        assert lv.scalar.blob.metadata == BlobMetadata(
            type=BlobType(
                format=format,
                dimensionality=BlobType.BlobDimensionality.SINGLE,
            )
        )
        assert lv.scalar.blob.uri is not None
    
        output = tf.to_python_value(ctx, lv, python_type)
        if isinstance(python_val, keras.Sequential):
            for p1, p2 in zip(output.weights, python_val.weights):
                np.testing.assert_array_equal(p1.numpy(), p2.numpy())
            assert True
        else:
>           assert isinstance(output, dict)
E           assert False
E            +  where False = isinstance(<keras.engine.functional.Functional object at 0x000001AE37320108>, dict)

Ahh yes, forgot to account for keras.Model in the check, after the refactor ....... just pushed fix now

[codecov]

Codecov Report

Merging #1242 (4fe5a31) into master (1fa73a0) will increase coverage by 0.02%.
The diff coverage is 72.78%.

@@            Coverage Diff             @@
##           master    #1242      +/-   ##
==========================================
+ Coverage   68.68%   68.71%   +0.02%     
==========================================
  Files         288      292       +4     
  Lines       26333    26482     +149     
  Branches     2486     2494       +8     
==========================================
+ Hits        18087    18196     +109     
- Misses       7768     7805      +37     
- Partials      478      481       +3     

Impacted Files	Coverage Δ
flytekit/extras/keras/__init__.py	0.00% <0.00%> (ø)
flytekit/extras/keras/native.py	38.09% <38.09%> (ø)
...flytekit/unit/extras/keras/test_transformations.py	93.22% <93.22%> (ø)
tests/flytekit/unit/extras/keras/test_native.py	97.29% <97.29%> (ø)
tests/flytekit/unit/cli/pyflyte/test_run.py	99.20% <0.00%> (ø)
tests/flytekit/unit/cli/pyflyte/workflow.py	100.00% <0.00%> (ø)
flytekit/clis/sdk_in_container/run.py	84.15% <0.00%> (+0.10%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.