Load performance test pipelines from files

[ckadner]

Description of your changes:

Updates to the performance test script:

• Move the the performance_test.py outside of the compiler folder (since the scope is beyond compiler testing)
• Add perf_test_config.yaml with paths of test pipelines
• Add a Status column to output CSV file
• Update compile_pyfile to remove loaded pipeline module after compilation to allow subsequent imports of another module with the same name

How to run it:

[kfp-tekton] $ NUM_WORKERS=10 sdk/python/tests/performance_tests.py

Environment variables:

  PUBLIC_IP:   169.12.34.56
  NAMESPACE:   None
  USER_INFO:   None
  CONNECT_SID: None
  NUM_WORKERS: 10
  TEST_CONFIG: sdk/python/tests/perf_test_config.yaml
  EXPERIMENT:  PERF_TEST
  OUTPUT_FILE: perf_test_20211201_130028_N10_169.12.34.56.csv
  OUTPUT_SEP:  ,

Loaded 5 pipelines from sdk/python/tests/perf_test_config.yaml

trusted-ai           failed     after 0:05:31 (21:00:29->21:06:00)
calc_pipeline        succeeded  after 0:11:10 (21:00:29->21:11:39)
Flip-Coin-Custom-Tas completed  after 0:11:30 (21:00:29->21:11:59)
Flip-Coin-samples    completed  after 0:14:14 (21:00:29->21:14:43)
withitem_nested      succeeded  after 0:18:01 (21:00:29->21:18:30)

Output:

Pipeline	Status	Compile_Time	Submit_Time	Run_Time	Num_TaskRuns	Num_Runs	Total_TaskRun_Time	Total_Run_Time
trusted-ai	Failed	0:00:00.370032	0:00:08.471185	0:05:27.214805	1	0	0:04:26	0:00:00
calc_pipeline	Succeeded	0:00:00.163394	0:00:08.852992	0:11:05.103077	3	0	0:08:07	0:00:00
Flip-Coin-Custom-Task	Completed	0:00:00.114080	0:00:09.203534	0:11:27.995799	3	2	0:08:11	0:00:00
Flip-Coin-samples	Completed	0:00:00.119328	0:00:08.908992	0:14:10.104801	7	0	0:12:36	0:00:00
withitem_nested	Succeeded	0:00:00.049208	0:00:08.116537	0:17:57.169451	1	1	0:04:20	0:17:27

Which issue is resolved by this Pull Request:

Resolves: #778

/cc @Tomcli
/cc @yhwang

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ckadner

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ckadner]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Tomcli]

thanks @ckadner, I also made few changes to the script mainly to update the examples to run for performance tests and fix some bugs. #795

[ckadner]

Since Python caches previsously loaded modules, this change was required to allow using the compile_pyfile() method multiple times within the same Python process and enable loading different pipeline scripts from files with the same ("module") name, i.e. we have multiple condition.py samples.

Without this change, we would get these errors:

ValueError trying to compile /Users/ckadner/PycharmProjects/kfp-tekton_ckadner/samples/flip-coin/condition.py: A function with @dsl.pipeline decorator is required in the py file.
ValueError trying to compile /Users/ckadner/PycharmProjects/kfp-tekton_ckadner/samples/flip-coin-custom-task/condition.py: A function with @dsl.pipeline decorator is required in the py file.

See: https://stackoverflow.com/questions/32234156/how-to-unimport-a-python-module-which-is-already-imported/57891909#57891909

[ckadner]

@Tomcli -- sorry for the concurrent PRs and difficult merging process. I hope by adding the changes you made to performance_tests.py here in my PR, we can merge this PR first and then merge your PR without the performance_tests.py changes to avoid messy merge conflict resolutions

[Tomcli]

/lgtm

[ckadner]

Thanks @Tomcli