Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load performance test pipelines from files #796

Conversation

ckadner
Copy link
Member

@ckadner ckadner commented Dec 1, 2021

Description of your changes:

Updates to the performance test script:

  • Move the the performance_test.py outside of the compiler folder (since the scope is beyond compiler testing)
  • Add perf_test_config.yaml with paths of test pipelines
  • Add a Status column to output CSV file
  • Update compile_pyfile to remove loaded pipeline module after compilation to allow subsequent imports of another module with the same name

How to run it:

[kfp-tekton] $ NUM_WORKERS=10 sdk/python/tests/performance_tests.py

Environment variables:

  PUBLIC_IP:   169.12.34.56
  NAMESPACE:   None
  USER_INFO:   None
  CONNECT_SID: None
  NUM_WORKERS: 10
  TEST_CONFIG: sdk/python/tests/perf_test_config.yaml
  EXPERIMENT:  PERF_TEST
  OUTPUT_FILE: perf_test_20211201_130028_N10_169.12.34.56.csv
  OUTPUT_SEP:  ,

Loaded 5 pipelines from sdk/python/tests/perf_test_config.yaml

trusted-ai           failed     after 0:05:31 (21:00:29->21:06:00)
calc_pipeline        succeeded  after 0:11:10 (21:00:29->21:11:39)
Flip-Coin-Custom-Tas completed  after 0:11:30 (21:00:29->21:11:59)
Flip-Coin-samples    completed  after 0:14:14 (21:00:29->21:14:43)
withitem_nested      succeeded  after 0:18:01 (21:00:29->21:18:30)

Output:

Pipeline Status Compile_Time Submit_Time Run_Time Num_TaskRuns Num_Runs Total_TaskRun_Time Total_Run_Time
trusted-ai Failed 0:00:00.370032 0:00:08.471185 0:05:27.214805 1 0 0:04:26 0:00:00
calc_pipeline Succeeded 0:00:00.163394 0:00:08.852992 0:11:05.103077 3 0 0:08:07 0:00:00
Flip-Coin-Custom-Task Completed 0:00:00.114080 0:00:09.203534 0:11:27.995799 3 2 0:08:11 0:00:00
Flip-Coin-samples Completed 0:00:00.119328 0:00:08.908992 0:14:10.104801 7 0 0:12:36 0:00:00
withitem_nested Succeeded 0:00:00.049208 0:00:08.116537 0:17:57.169451 1 1 0:04:20 0:17:27

Which issue is resolved by this Pull Request:

Resolves: #778

/cc @Tomcli
/cc @yhwang

@ckadner ckadner added area/testing area/sdk python Pull requests that update Python code labels Dec 1, 2021
@ckadner ckadner added this to the performance_script_TODOs milestone Dec 1, 2021
@ckadner ckadner requested review from yhwang and Tomcli December 1, 2021 18:13
@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ckadner

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ckadner ckadner force-pushed the load_pipelines_from_files_for_performance_test branch from bb10915 to 0d05b2a Compare December 1, 2021 18:18
@Tomcli
Copy link
Member

Tomcli commented Dec 1, 2021

thanks @ckadner, I also made few changes to the script mainly to update the examples to run for performance tests and fix some bugs. #795

* Add perf_test_config.yaml with paths of test pipelines
* Add a 'Status' column to output CSV file
* Update compile_pyfile to remove loaded pipeline module after
  compilation to allow subsequent imports of another module
  with the same name

Resolves kubeflow#778

Signed-off-by: Christian Kadner <[email protected]>
@ckadner ckadner force-pushed the load_pipelines_from_files_for_performance_test branch from 0d05b2a to acf623e Compare December 1, 2021 21:22
@@ -82,6 +82,7 @@ def compile_pyfile(pyfile, function_name, output_path, type_check, tekton_pipeli
_compile_pipeline_function(pipeline_funcs, function_name, output_path, type_check, tekton_pipeline_conf)
finally:
del sys.path[0]
sys.modules.pop(os.path.splitext(filename)[0])
Copy link
Member Author

@ckadner ckadner Dec 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since Python caches previsously loaded modules, this change was required to allow using the compile_pyfile() method multiple times within the same Python process and enable loading different pipeline scripts from files with the same ("module") name, i.e. we have multiple condition.py samples.

Without this change, we would get these errors:

ValueError trying to compile /Users/ckadner/PycharmProjects/kfp-tekton_ckadner/samples/flip-coin/condition.py: A function with @dsl.pipeline decorator is required in the py file.
ValueError trying to compile /Users/ckadner/PycharmProjects/kfp-tekton_ckadner/samples/flip-coin-custom-task/condition.py: A function with @dsl.pipeline decorator is required in the py file.

See: https://stackoverflow.com/questions/32234156/how-to-unimport-a-python-module-which-is-already-imported/57891909#57891909

@ckadner ckadner requested a review from Tomcli December 1, 2021 21:31
@ckadner
Copy link
Member Author

ckadner commented Dec 1, 2021

@Tomcli -- sorry for the concurrent PRs and difficult merging process. I hope by adding the changes you made to performance_tests.py here in my PR, we can merge this PR first and then merge your PR without the performance_tests.py changes to avoid messy merge conflict resolutions

@Tomcli
Copy link
Member

Tomcli commented Dec 1, 2021

/lgtm

@google-oss-prow google-oss-prow bot added the lgtm label Dec 1, 2021
@google-oss-prow google-oss-prow bot merged commit 2c61c8a into kubeflow:master Dec 1, 2021
@ckadner
Copy link
Member Author

ckadner commented Dec 1, 2021

Thanks @Tomcli

wzhanw pushed a commit to wzhanw/kfp-tekton that referenced this pull request Mar 22, 2022
* Add perf_test_config.yaml with paths of test pipelines
* Add a 'Status' column to output CSV file
* Update compile_pyfile to remove loaded pipeline module after
  compilation to allow subsequent imports of another module
  with the same name

Resolves kubeflow#778

Signed-off-by: Christian Kadner <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make performance script to load pipelines in any python file
2 participants