Use wheels for Dataflow postcommit tests #25970

Abacn · 2023-03-24T20:46:15Z

Fixes #25966
Fixes #26049

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2023-03-24T21:32:33Z

Codecov Report

Merging #25970 (d41b7c6) into master (87db1ae) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #25970      +/-   ##
==========================================
- Coverage   71.17%   71.17%   -0.01%     
==========================================
  Files         787      787              
  Lines      103294   103311      +17     
==========================================
+ Hits        73523    73532       +9     
- Misses      28283    28291       +8     
  Partials     1488     1488

Flag	Coverage Δ
python	`79.85% <ø> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 8 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Abacn · 2023-03-25T12:11:56Z

Tests on dataflow actually passed and indeed takes much faster: https://ci-beam.apache.org/job/beam_PostCommit_Python310_PR/38/

However, flink runner test not running due to [Errno 2] No such file or directory: 'apache-beam-2.47.0.dev0/apache_beam/... this is a racing condition that sdist generates a temporary directory during packaging and delete it when finishing.

-- this is due to pypa/setuptools#1222

Abacn · 2023-03-25T14:24:03Z

Run Python 3.10 PostCommit

Update: running time from ~3h10min (cron job) reduced to 2.5 h (this PR)

Abacn · 2023-03-27T21:34:42Z

Run Python 3.7 PostCommit

Abacn · 2023-03-27T21:34:57Z

Run Python 3.8 PostCommit

Abacn · 2023-03-28T00:18:03Z

Run Python 3.9 PostCommit

Abacn · 2023-03-28T03:56:04Z

Python3.7 PostCommit passed
Python3.8 PostCommit fails on :sdks:python:test-suites:dataflow:py38:tensorRTtests
due to that the wheel is not installed (get silently ignored) on the custom SDK container at us.gcr.io/apache-beam-testing/python-postcommit-it/tensor_rt:latest. The reason is that The container was custom build ([Task]: Build docker image for tensor RT tests during Run time and also provide Python 3.7, 3.9 Docker containers. #22651 (comment)) and is half year old and used Beam 2.40.0, where the suffix validation for Beam wheel has since changed (Update staging of Python wheels #24114).

Short term solution is that still feed the test with tarball for now. When [Task]: Build docker image for tensor RT tests during Run time and also provide Python 3.7, 3.9 Docker containers. #22651 is done it should be able to enable wheel sdk_location for tensorRTtests.

Python3.9 PostCommit fails unrelated:

FAILED 
apache_beam/ml/inference/tensorflow_inference_it_test.py::TensorflowInference::test_tf_imagenet_image_segmentation - 
AssertionError: 'Background' != 'Military Uniform'

this test also fails on latest master: https://ci-beam.apache.org/job/beam_PostCommit_Python39/1630/consoleFull

Python3.10 PostCommit failed earlier at apache_beam.ml.inference.sklearn_inference_it_test.SklearnInference.test_sklearn_regression due to Force version of sklearn in examples #25968 should fixed on latest master

Abacn · 2023-03-31T17:32:48Z

Run Python 3.8 PostCommit

Abacn · 2023-03-31T17:32:55Z

Run Python 3.10 PostCommit

Abacn · 2023-03-31T20:44:07Z

Thanks @tvalentyn for the suggestion. I have rewritten the part of build wheel. Summary of change since last review:

added bdistPy{version}{platfoem} tasks in :sdks:python gradle project to build wheels.
created a 'initializeForDataflowJob' task in test-suites/dataflow/common.gradle handling sdk distribution setup.
reverted changes to BeamModulePlugin.

PostCommit succeeded. Ready for review again.

Abacn · 2023-04-04T17:20:59Z

gradle.properties

 flink_versions=1.12,1.13,1.14,1.15,1.16
+# supported python versions
+python_versions=3.7,3.8,3.9,3.10


Added a gradle property here because now :sdks:python project need it to define bdist tasks. This also simplifies gradle.properties in python/test-suites a little bit because now it has a single source of truth for supported python version

Can we use this one instead?

beam/.test-infra/jenkins/PythonTestProperties.groovy

Line 22 in dced773

final static List<String> ALL_SUPPORTED_VERSIONS = [

Thanks for pointing to this. Not aware of it. This code path is for Jenkins only (.test-infra/jenkins/); gradle files works independently with Jenkins (gradle tasks can trigger locally). To reduce to duplicate declaration, refer to gradle.properties in Jenkins script should work

there is also https://github.com/apache/beam/blob/master/sdks/python/test-suites/gradle.properties Would be good to see if we can these version configurations together.

is https://github.com/apache/beam/blob/master/sdks/python/test-suites/gradle.properties accessible from the context where you need this?

The problem is sdks/python/test-suites/gradle.properties is not accessible to sdks/python gradle project. I understand test-suites/gradle.properties as a configuration to manage which python version used in different test. While for bdist it belongs to the python root project (just as sdist). I did not find a source of truth for supported Python versions accessible in the sdks/python context.

ack, thanks for checking. @AnandInguva FYI one more place to add python versions in the future.

Thanks, will experiment to read gradle.properties in jenkins/github actions so reduce the duplicate declarations of py version

tvalentyn · 2023-04-04T21:37:00Z

sdks/python/build.gradle

+          environment CIBW_BUILD: "cp${pyversion}-${idsuffix}"
+          environment CIBW_ENVIRONMENT: "SETUPTOOLS_USE_DISTUTILS=stdlib"
+          environment CIBW_BEFORE_BUILD: "pip install cython numpy && pip install --upgrade setuptools"
+          // note: sync cibuildwheel version with GitHub Action build_wheel


lets add a pointer to exact place where this needs to be modified and maybe make it a inline comment, and do the same in the counterpart where wheels are built.

thanks, added to both build.gradle and .yml

Abacn · 2023-04-05T20:07:08Z

Doing a rebase, will squash the reviewed commit (that is before 8329ee5)

tvalentyn · 2023-04-05T23:17:58Z

Reminder about code freeze: https://lists.apache.org/thread/rpmq9r9tbgrry8yco70mzyt4psyk5m20

tvalentyn · 2023-04-05T23:18:13Z

Thank you very much, @Abacn !

Abacn · 2023-04-06T03:13:13Z

Run Python 3.7 PostCommit

Abacn · 2023-04-06T03:15:24Z

Run Python 3.11 PostCommit

Abacn · 2023-04-06T21:37:46Z

Jenkins not triggering. The latest commit just switched to opt-in buildWheels from opt-out (convenient for testing without the need of triggering seed job). Merging based on the signal in 697635a.

at there Python postcommit passed. "pending" tests were actually passed; failed Java tests irrelevant.

Abacn · 2023-04-06T23:54:35Z

There are 3 stuck Java test and 1 failed java tests unrelated. Other precommits passed. Merging for now.

github-actions bot added build docker python labels Mar 24, 2023

Abacn force-pushed the usewheel branch from 0697848 to d3e4ba9 Compare March 24, 2023 21:07

Abacn force-pushed the usewheel branch 2 times, most recently from 53f64a9 to e50b97e Compare March 24, 2023 22:10

github-actions bot removed the docker label Mar 24, 2023

Abacn force-pushed the usewheel branch 7 times, most recently from 008170e to 2a4e5a3 Compare March 25, 2023 02:16

Use wheels for Dataflow postcommit tests

e9cffe8

Abacn force-pushed the usewheel branch 2 times, most recently from fff0bf3 to e783586 Compare March 25, 2023 04:25

PoC only - if build wheel locally works

29c65bb

Abacn force-pushed the usewheel branch from e783586 to 29c65bb Compare March 25, 2023 04:27

Abacn mentioned this pull request Mar 25, 2023

[Bug]: Cythonization is compiled with language level 2 #25979

Closed

15 tasks

Resolve racing condition concurrently run sdist

bb766cc

Abacn mentioned this pull request Mar 27, 2023

[Task]: Pass sdk wheel instead of tarball for Python Dataflow PostCommit #25966

Closed

15 tasks

Merge branch 'master' into usewheel

a91ab4f

github-actions bot removed the build label Mar 31, 2023

Abacn force-pushed the usewheel branch from 1b93252 to d8a8d14 Compare March 31, 2023 15:48

Abacn mentioned this pull request Apr 4, 2023

[Task]: Run some BigQuery integration tests only on one Dataflow Python3.X PostCommit #23573

Closed

Abacn commented Apr 4, 2023

View reviewed changes

tvalentyn reviewed Apr 4, 2023

View reviewed changes

Abacn force-pushed the usewheel branch from bf29864 to 012c6a2 Compare April 5, 2023 18:20

github-actions bot added the build label Apr 5, 2023

Abacn added 3 commits April 5, 2023 16:09

Build wheel using cibuildwheels

f936ecd

address comment - pointer to exact place

52d1134

Merge branch 'master' into usewheel

697635a

Abacn force-pushed the usewheel branch from 012c6a2 to 697635a Compare April 5, 2023 20:14

tvalentyn approved these changes Apr 5, 2023

View reviewed changes

damccorm force-pushed the master branch from 2f1e647 to 87db1ae Compare April 6, 2023 14:15

Change back to opt-in before merging

d41b7c6

Abacn merged commit 77b7b4e into apache:master Apr 6, 2023

Abacn deleted the usewheel branch April 6, 2023 23:55

Abacn mentioned this pull request Apr 7, 2023

Use gradle property supported Python version in .test-infra #26171

Closed

3 tasks

Abacn mentioned this pull request May 3, 2023

Add streaming tests to Storage Write API Python wrapper #26491

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use wheels for Dataflow postcommit tests #25970

Use wheels for Dataflow postcommit tests #25970

Abacn commented Mar 24, 2023 •

edited

Loading

codecov bot commented Mar 24, 2023 •

edited

Loading

Abacn commented Mar 25, 2023 •

edited

Loading

Abacn commented Mar 25, 2023 •

edited

Loading

Abacn commented Mar 27, 2023

Abacn commented Mar 27, 2023

Abacn commented Mar 28, 2023

Abacn commented Mar 28, 2023

Abacn commented Mar 31, 2023

Abacn commented Mar 31, 2023

Abacn commented Mar 31, 2023

Abacn Apr 4, 2023

AnandInguva Apr 4, 2023

Abacn Apr 4, 2023

tvalentyn Apr 4, 2023 •

edited

Loading

tvalentyn Apr 4, 2023

Abacn Apr 4, 2023

tvalentyn Apr 5, 2023

Abacn Apr 6, 2023

tvalentyn Apr 4, 2023

Abacn Apr 5, 2023

Abacn commented Apr 5, 2023

tvalentyn commented Apr 5, 2023

tvalentyn commented Apr 5, 2023

Abacn commented Apr 6, 2023

Abacn commented Apr 6, 2023

Abacn commented Apr 6, 2023

Abacn commented Apr 6, 2023

Use wheels for Dataflow postcommit tests #25970

Use wheels for Dataflow postcommit tests #25970

Conversation

Abacn commented Mar 24, 2023 • edited Loading

GitHub Actions Tests Status (on master branch)

codecov bot commented Mar 24, 2023 • edited Loading

Codecov Report

Abacn commented Mar 25, 2023 • edited Loading

Abacn commented Mar 25, 2023 • edited Loading

Abacn commented Mar 27, 2023

Abacn commented Mar 27, 2023

Abacn commented Mar 28, 2023

Abacn commented Mar 28, 2023

Abacn commented Mar 31, 2023

Abacn commented Mar 31, 2023

Abacn commented Mar 31, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tvalentyn Apr 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Abacn commented Apr 5, 2023

tvalentyn commented Apr 5, 2023

tvalentyn commented Apr 5, 2023

Abacn commented Apr 6, 2023

Abacn commented Apr 6, 2023

Abacn commented Apr 6, 2023

Abacn commented Apr 6, 2023

Abacn commented Mar 24, 2023 •

edited

Loading

codecov bot commented Mar 24, 2023 •

edited

Loading

Abacn commented Mar 25, 2023 •

edited

Loading

Abacn commented Mar 25, 2023 •

edited

Loading

tvalentyn Apr 4, 2023 •

edited

Loading