Attempt to bump python version to 3.11 #2417

noklam · 2023-03-13T15:13:42Z

Signed-off-by: Nok Chan [email protected]

NOTE: Kedro datasets are moving from kedro.extras.datasets to a separate kedro-datasets package in
kedro-plugins repository. Any changes to the dataset implementations
should be done by opening a pull request in that repository.

Description

See what's failing in CI and estimate the effort of actually supporting newer version.

Investigation of #2048 #2270

Development notes

Checklist

Read the contributing guidelines
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes

noklam · 2023-03-21T11:29:13Z

Note:
I need to manually trigger a "Nightly build" on CI to build Python images for 3.11

noklam · 2023-03-22T15:20:33Z

Notes: (will keep this updated)
Challenges

Example 1 - Tensorflow haven't supported Python 3.11
- Do we skip all the related datasets on a specific Python version? If so, how?
- Follow up - how do we skip tensorflow install only in Python 3.11?

noklam · 2023-03-22T16:32:24Z

Notes:

=========================== short test summary info ============================
SKIPPED [1] tests/extras/datasets/video/test_video_objects.py:125: Can't deal with videos with missing time info
ERROR tests/extras/datasets/dask/test_parquet_dataset.py
ERROR tests/extras/datasets/matplotlib/test_matplotlib_writer.py
ERROR tests/extras/datasets/spark/test_spark_dataset.py
ERROR tests/extras/datasets/video/test_video_dataset.py
ERROR tests/io/test_incremental_dataset.py
ERROR tests/io/test_partitioned_dataset.py
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_cant_pickle - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_read_existing_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_overwrite_empty_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_overwrite_not_empty_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_insert_not_empty_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_deltatable_dataset.py::TestDeltaTableDataSet::test_load - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_upsert_config_err - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_deltatable_dataset.py::TestDeltaTableDataSet::test_save - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_upsert_empty_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_deltatable_dataset.py::TestDeltaTableDataSet::test_exists - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_upsert_not_empty_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_invalid_pk_provided - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_save_and_load[None-None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_invalid_write_mode_provided - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_load_missing_model[None-None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_exists[None-None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_hdf5_save_format - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_unused_subclass_model_hdf5_save_format - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_protocol_usage[s3://bucket/test_tf-S3FileSystem] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_protocol_usage[file:///tmp/test_tf-LocalFileSystem] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_protocol_usage[/tmp/test_tf-LocalFileSystem] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_protocol_usage[gcs://bucket/test_tf-GCSFileSystem] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_protocol_usage[https://example.com/test_tf-HTTPFileSystem] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_load_extra_params[None-None-load_args0] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_catalog_release - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_fs_args[fs_args0] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_exists_with_exception[None-None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_save_and_overwrite_existing_model[None-None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_save_and_load[2019-01-01T23.59.59.999Z-2019-01-01T23.59.59.999Z] - ModuleNotFoundError: No module named 'tensorflow'
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_save_and_load[None-None] - ModuleNotFoundError: No module named 'tensorflow'
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_hdf5_save_format[None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_prevent_overwrite[None-None] - ModuleNotFoundError: No module named 'tensorflow'
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_save_version_warning[2019-01-01T23.59.59.999Z-2019-01-02T00.00.00.000Z] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_http_filesystem_no_versioning - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_exists[None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_invalid_schema_insert - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_no_versions[None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_version_str_repr[None-None-None-None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_versioning_existing_dataset[None-None-None-None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_save_and_load_with_device[None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_insert_to_non_existent_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_read_from_non_existent_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_save_delta_format - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_memory_dataset.py::test_load_modify_original_data - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_memory_dataset.py::test_save_modify_original_data - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_memory_dataset.py::test_load_returns_same_spark_object - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_memory_dataset.py::test_str_representation - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
FAILED tests/extras/datasets/holoviews/test_holoviews_writer.py::TestHoloviewsWriter::test_save_data[None-None] - kedro.io.core.DataSetError: Failed while saving data to data set HoloviewsWriter(filepath=/tmp/pytest-of-circleci/pytest-0/popen-gw0/test_save_data_None_None_0/test.png, protocol=file, save_args={'fmt': png}).
Could not find a 'matplotlib' renderer, available renderers are: .
FAILED tests/extras/datasets/holoviews/test_holoviews_writer.py::TestHoloviewsWriter::test_exists[None-None] - kedro.io.core.DataSetError: Failed while saving data to data set HoloviewsWriter(filepath=/tmp/pytest-of-circleci/pytest-0/popen-gw0/test_exists_None_None_0/test.png, protocol=file, save_args={'fmt': png}).
Could not find a 'matplotlib' renderer, available renderers are: .
FAILED tests/extras/datasets/holoviews/test_holoviews_writer.py::TestHoloviewsWriterVersioned::test_prevent_overwrite[None-None] - kedro.io.core.DataSetError: Failed while saving data to data set HoloviewsWriter(filepath=/tmp/pytest-of-circleci/pytest-0/popen-gw0/test_prevent_overwrite_None_No0/test.png, protocol=file, save_args={'fmt': png}, version=Version(load=None, save='2023-03-22T16.02.40.136Z')).
Could not find a 'matplotlib' renderer, available renderers are: .
FAILED tests/extras/datasets/holoviews/test_holoviews_writer.py::TestHoloviewsWriterVersioned::test_save_version_warning[2019-01-02T00.00.00.000Z-2019-01-01T23.59.59.999Z] - kedro.io.core.DataSetError: Failed while saving data to data set HoloviewsWriter(filepath=/tmp/pytest-of-circleci/pytest-0/popen-gw0/test_save_version_warning_20190/test.png, protocol=file, save_args={'fmt': png}, version=Version(load='2019-01-01T23.59.59.999Z', save='2019-01-02T00.00.00.000Z')).
Could not find a 'matplotlib' renderer, available renderers are: .
FAILED tests/extras/datasets/holoviews/test_holoviews_writer.py::TestHoloviewsWriterVersioned::test_exists[None-None] - kedro.io.core.DataSetError: Failed while saving data to data set HoloviewsWriter(filepath=/tmp/pytest-of-circleci/pytest-0/popen-gw0/test_exists_None_None_1/test.png, protocol=file, save_args={'fmt': png}, version=Version(load=None, save='2023-03-22T16.02.40.556Z')).
Could not find a 'matplotlib' renderer, available renderers are: .
FAILED tests/extras/datasets/holoviews/test_holoviews_writer.py::TestHoloviewsWriterVersioned::test_save_data[None-None] - kedro.io.core.DataSetError: Failed while saving data to data set HoloviewsWriter(filepath=/tmp/pytest-of-circleci/pytest-0/popen-gw0/test_save_data_None_None_1/test.png, protocol=file, save_args={'fmt': png}, version=Version(load=None, save='2023-03-22T16.02.40.740Z')).
Could not find a 'matplotlib' renderer, available renderers are: .
FAILED tests/extras/datasets/holoviews/test_holoviews_writer.py::TestHoloviewsWriterVersioned::test_versioning_existing_dataset[None-None-None-None] - kedro.io.core.DataSetError: Failed while saving data to data set HoloviewsWriter(filepath=/tmp/pytest-of-circleci/pytest-0/popen-gw0/test_versioning_existing_datas0/test.png, protocol=file, save_args={'fmt': png}).
Could not find a 'matplotlib' renderer, available renderers are: .
FAILED tests/framework/cli/test_cli_hooks.py::TestKedroCLIHooks::test_kedro_cli_should_invoke_cli_hooks_from_plugin[-V-0] - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/framework/cli/test_cli.py::TestKedroCLI::test_project_commands_no_clipy - ModuleNotFoundError
FAILED tests/framework/cli/test_cli_hooks.py::TestKedroCLIHooks::test_kedro_cli_should_invoke_cli_hooks_from_plugin[info-2] - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/framework/cli/test_cli.py::TestKedroCLI::test_project_commands_invalid_clipy - AttributeError: None does not have the attribute '_is_project'
FAILED tests/framework/cli/test_cli_hooks.py::TestKedroCLIHooks::test_kedro_cli_should_invoke_cli_hooks_from_plugin[pipeline list-2] - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/framework/cli/test_cli.py::TestKedroCLI::test_project_commands_valid_clipy - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/framework/cli/test_cli.py::TestKedroCLI::test_kedro_cli_with_project - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/framework/cli/test_cli_hooks.py::TestKedroCLIHooks::test_kedro_cli_should_invoke_cli_hooks_from_plugin[starter-0] - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/io/test_data_catalog.py::TestDataCatalogVersioned::test_no_versions_with_cloud_protocol - AssertionError: Regex pattern did not match.
 Regex: "Did\\ not\\ find\\ any\\ versions\\ for\\ CSVDataSet\\(filepath=bucket/file\\.csv,\\ protocol=\\{\\},\\ save_args=\\{'index':\\ False\\},\\ version=Version\\(load=None,\\ save=None\\)\\)\\.\\ This\\ could\\ be\\ due\\ to\\ insufficient\\ permission\\."
 Input: "Failed while loading data from data set CSVDataSet(filepath=bucket/file.csv, protocol={}, save_args={'index': False}, version=Version(load=None, save=None)).\nUnable to locate credentials"
FAILED tests/tools/test_cli.py::TestCLITools::test_get_cli_structure_raw - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/tools/test_cli.py::TestCLITools::test_get_cli_structure_depth - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/tools/test_cli.py::TestCLITools::test_get_cli_structure_help - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'

astrojuanlu · 2023-05-11T07:15:15Z

Rebased this on top of main and attempted to include Spark and TensorFlow.

astrojuanlu · 2023-05-11T08:52:02Z

There seem to be some problems now with delta-spark:

Despite loosening the upper bound, pip-compile still errors on 3.11:

The conflict is caused by:
    The user requested pyspark<4.0 and >=3.4
    delta-spark 2.3.0 depends on pyspark<3.4.0 and >=3.3.0
    The user requested pyspark<4.0 and >=3.4
    delta-spark 2.2.0 depends on pyspark<3.4.0 and >=3.3.0
    The user requested pyspark<4.0 and >=3.4
    delta-spark 2.1.1 depends on pyspark<3.4.0 and >=3.3.0
    The user requested pyspark<4.0 and >=3.4
    delta-spark 2.1.0 depends on pyspark<3.4.0 and >=3.3.0
    The user requested pyspark<4.0 and >=3.4
    delta-spark 2.0.2 depends on pyspark<3.3.0 and >=3.2.0
    The user requested pyspark<4.0 and >=3.4
    delta-spark 2.0.1 depends on pyspark<3.3.0 and >=3.2.0
    The user requested pyspark<4.0 and >=3.4
    delta-spark 2.0.0 depends on pyspark<3.3.0 and >=3.2.0
    The user requested pyspark<4.0 and >=3.4
    delta-spark 1.2.1 depends on pyspark<3.3.0 and >=3.2.0

delta-spark merged support for Spark 3.4 one week ago delta-io/delta@5c3f4d3 so the next released version should be compatible.

In the other Python versions, it's leading to 2 tests failing: TestDeltaTableDataSet.test_save and TestDeltaTableDataSet.test_exists.

We're very close!

astrojuanlu · 2023-05-25T22:12:59Z

https://pypi.org/project/delta-spark/2.4.0/

Signed-off-by: Nok <[email protected]>

Signed-off-by: SajidAlamQB <[email protected]>

Signed-off-by: Nok <[email protected]>

…spark3.4 Signed-off-by: Nok <[email protected]>

noklam · 2023-07-03T15:16:08Z

kedro/extras/datasets/spark/spark_dataset.py

+            if hasattr(exception, "desc"):  # Pyspark <3.4
+                message = exception.desc
+            else:
+                message = exception.message
+
+            if "Path does not exist:" in message or "is not a Delta table" in message:


In theory we should freeze it, but this minor change would allow us release Python 3.11 without breaking change. We should also have this change in kedro-datasets

Signed-off-by: Nok <[email protected]>

Signed-off-by: SajidAlamQB <[email protected]>

sbrugman · 2023-07-20T11:48:12Z

By updating the classifiers in pyproject.toml, pip will allow users to install from py3.11 after the release:

classifiers = [
    "Development Status :: 4 - Beta",
    "Programming Language :: Python :: 3.7",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
]

sbrugman · 2023-07-20T11:48:46Z

Any help needed here? Would be great to be able to bump versions.. Python 3.7 is also officially end-of-life and can be dropped

astrojuanlu · 2023-08-14T09:59:10Z

Superseded by #2851

noklam changed the title ~~Attempt to bump python version~~ Attempt to bump python version to 3.11 Mar 17, 2023

noklam mentioned this pull request Mar 22, 2023

Kedro-dataset release process kedro-org/kedro-plugins#405

Open

astrojuanlu mentioned this pull request May 10, 2023

Support python 3.11 #2270

Closed

astrojuanlu force-pushed the bump/python311 branch from 548965a to 412ba84 Compare May 11, 2023 07:14

astrojuanlu force-pushed the bump/python311 branch 2 times, most recently from 2be0176 to edf3604 Compare May 11, 2023 08:30

noklam added 18 commits May 26, 2023 14:29

update CI

8265db7

Signed-off-by: Nok <[email protected]>

Release all the upper bounds

02ea106

Signed-off-by: Nok <[email protected]>

bump more versions

c2d1bd5

Signed-off-by: Nok <[email protected]>

update requirements

29e5cbf

Signed-off-by: Nok <[email protected]>

update requirements

34ce2bf

Signed-off-by: Nok <[email protected]>

add dependency

249fded

Signed-off-by: Nok <[email protected]>

update scipy

4e03227

Signed-off-by: Nok <[email protected]>

fix typo

121a1be

Signed-off-by: Nok <[email protected]>

update

6aa27b0

Signed-off-by: Nok <[email protected]>

revert all changes

6318f25

Signed-off-by: Nok <[email protected]>

bump holoviews

aa53b24

Signed-off-by: Nok <[email protected]>

release pyarrow

2ffb41c

Signed-off-by: Nok <[email protected]>

relax scipy

cca35a2

Signed-off-by: Nok <[email protected]>

bump tensorflow

483cf0c

Signed-off-by: Nok <[email protected]>

give up tensorflow, it's not supported yet

11f4db5

Signed-off-by: Nok <[email protected]>

bump lower bound of pyarrow as it caused build error

ed19a95

Signed-off-by: Nok <[email protected]>

bump pyarrow lower bound

4b5ca3d

Signed-off-by: Nok <[email protected]>

bump spark version

14896f1

Signed-off-by: Nok <[email protected]>

SajidAlamQB and others added 3 commits June 29, 2023 14:36

make moto fixed for 3.10

d855426

Signed-off-by: SajidAlamQB <[email protected]>

fix spark compatibility issue

bc6f09c

Signed-off-by: Nok <[email protected]>

Fix Spark error handling in exists() so it will be compatible with py…

3843caf

…spark3.4 Signed-off-by: Nok <[email protected]>

noklam commented Jul 3, 2023

View reviewed changes

noklam added 9 commits July 3, 2023 15:18

make delta compatible in the same fashion as spark

da40d8c

Signed-off-by: Nok <[email protected]>

refactor the test to pass coverage

ba47ba7

Signed-off-by: Nok <[email protected]>

fix pyspark test

3997aa8

Signed-off-by: Nok <[email protected]>

fix test that are too strict

7806129

Signed-off-by: Nok <[email protected]>

fix delta

66d3966

Signed-off-by: Nok <[email protected]>

refactor jdbc dataset tests with pytest and bug fix

761d008

Signed-off-by: Nok <[email protected]>

fix issues with spark test for different version, introduce semver

e8479db

Signed-off-by: Nok <[email protected]>

Merge branch 'main' into bump/python311

3e86cd1

fix linter

bdea52d

Signed-off-by: Nok <[email protected]>

noklam mentioned this pull request Jul 5, 2023

Allow any version of Python (including 3.11) in metadata #2742

Merged

5 tasks

SajidAlamQB added 10 commits July 14, 2023 09:08

Merge branch 'main' into bump/python311

e622c8c

add test for covering getOrCreate

18eaaed

Signed-off-by: SajidAlamQB <[email protected]>

more specific with test

7af8483

Signed-off-by: SajidAlamQB <[email protected]>

add 3.11 support to metadata

009742a

Signed-off-by: SajidAlamQB <[email protected]>

Merge branch 'main' into bump/python311

f4e0839

check getOrCreate() method is not None

3435950

Signed-off-by: SajidAlamQB <[email protected]>

Update deltatable_dataset.py

82b80fc

Signed-off-by: SajidAlamQB <[email protected]>

revert deltatable change

8d51f8b

Signed-off-by: SajidAlamQB <[email protected]>

Update test_spark_jdbc_dataset.py

7ccad55

Signed-off-by: SajidAlamQB <[email protected]>

fix deltatable

70f12be

Signed-off-by: SajidAlamQB <[email protected]>

astrojuanlu mentioned this pull request Jul 27, 2023

Add Python 3.11 support to Kedro #2851

Merged

5 tasks

astrojuanlu closed this Aug 14, 2023

astrojuanlu deleted the bump/python311 branch August 14, 2023 09:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Attempt to bump python version to 3.11 #2417

Attempt to bump python version to 3.11 #2417

noklam commented Mar 13, 2023 •

edited

Loading

noklam commented Mar 21, 2023

noklam commented Mar 22, 2023

noklam commented Mar 22, 2023 •

edited

Loading

astrojuanlu commented May 11, 2023

astrojuanlu commented May 11, 2023

astrojuanlu commented May 25, 2023

noklam Jul 3, 2023

sbrugman commented Jul 20, 2023

sbrugman commented Jul 20, 2023

astrojuanlu commented Aug 14, 2023

Attempt to bump python version to 3.11 #2417

Attempt to bump python version to 3.11 #2417

Conversation

noklam commented Mar 13, 2023 • edited Loading

Description

Development notes

Checklist

noklam commented Mar 21, 2023

noklam commented Mar 22, 2023

noklam commented Mar 22, 2023 • edited Loading

astrojuanlu commented May 11, 2023

astrojuanlu commented May 11, 2023

astrojuanlu commented May 25, 2023

noklam Jul 3, 2023

Choose a reason for hiding this comment

sbrugman commented Jul 20, 2023

sbrugman commented Jul 20, 2023

astrojuanlu commented Aug 14, 2023

noklam commented Mar 13, 2023 •

edited

Loading

noklam commented Mar 22, 2023 •

edited

Loading