Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmarking automation with on-demand CI test #3335

Merged
merged 57 commits into from
Jun 29, 2021

Conversation

axsaucedo
Copy link
Contributor

@axsaucedo axsaucedo commented Jun 26, 2021

This PR contains:

  • Extension of autoamted Argo benchmarking to perform parameter combinatiosn in the workflow itself
  • Also parallel workflows can be configured to speed up the length of the benchmarking
  • Added integration test that currently only configures a basic benchmark but can be extended to specific checks
  • Added jenkins X job with 4 CPU to ensure
  • Ensure notebook tests (incl this new one) actually print the output
  • Prints results back as comment to jx - example below

image

Link to sample comment: #3335 (comment)

Some context pieces to consider:

  • Unfortunately github actions was not possible as there's a limit of 2 CPU on workers, and cannot use docker with KIND so all CLIs have to be installed in every run (ie kubectl, helm, etc)

Outstanding work [to be done in separate PR]:

@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@axsaucedo
Copy link
Contributor Author

/test benchmark

@axsaucedo
Copy link
Contributor Author

/test benchmark

@axsaucedo
Copy link
Contributor Author

/test benchmark

@axsaucedo axsaucedo changed the title Benchmarking automation with on-demand CI test WIP: Benchmarking automation with on-demand CI test Jun 26, 2021
@axsaucedo
Copy link
Contributor Author

/test benchmark

@axsaucedo
Copy link
Contributor Author

/test benchmark

@axsaucedo
Copy link
Contributor Author

/test benchmark

1 similar comment
@axsaucedo
Copy link
Contributor Author

/test benchmark

@axsaucedo
Copy link
Contributor Author

/test benchmark

Copy link
Contributor

@RafalSkolasinski RafalSkolasinski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm


- name: run-benchmark-iteration-step-template
inputs:
parameters:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think listing the parameters here again is required - I have the feeling these can be just listed once globally for the workflow.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unfortunately it is needed, each of the sections is like a "function", so you have to define the arguemnts, and then pass them, at every level

@@ -0,0 +1,495 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Side remark - as we know now this are also required for GKE cluster.


Reply via ReviewNB

@@ -0,0 +1,495 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line #11.        --set seldonDeployment.apiType="rest|grpc" \

If the value is defined as A|B will it then create two workflows?


Reply via ReviewNB

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly, you can define mutliple values and it will generate all the different combinations and run the equivallent number of jobs

@seldondev
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RafalSkolasinski

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@axsaucedo
Copy link
Contributor Author

/test benchmark

@axsaucedo
Copy link
Contributor Author

/test benchmark

@seldondev
Copy link
Collaborator

Benchmark results

mean 50th 90th 95th 99th throughputAchieved success errors name replicas serverWorkers serverThreads modelUri image server apiType requestsCpu requestsMemory limitsCpu limitsMemory benchmarkCpu concurrency duration rate disableOrchestrator
0 6.83401 6.56718 8.25801 9.04552 11.7743 146.248 4389 0 seldon-batch-job-sdep-1 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER rest 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 true
1 5.59861 5.35421 6.96334 7.77089 10.3612 172.17 5166 1 seldon-batch-job-sdep-3 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER grpc 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 true
2 5.58779 5.35659 6.98933 7.67698 10.1868 172.484 5176 1 seldon-batch-job-sdep-2 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER grpc 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
3 6.8238 6.56463 8.22992 9.03618 11.9978 146.483 4396 0 seldon-batch-job-sdep-0 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER rest 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false

@axsaucedo
Copy link
Contributor Author

/test benchmark

@seldondev
Copy link
Collaborator

Benchmark results

mean 50th 90th 95th 99th throughputAchieved success errors name replicas serverWorkers serverThreads modelUri image server apiType requestsCpu requestsMemory limitsCpu limitsMemory benchmarkCpu concurrency duration rate disableOrchestrator
0 6.47571 6.19014 7.81799 8.64152 11.8734 154.36 4633 0 seldon-batch-job-sdep-0 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER rest 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
1 6.48343 6.23287 7.7959 8.51098 11.0096 154.176 4626 0 seldon-batch-job-sdep-1 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER rest 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 true
2 5.46715 5.15307 6.98203 7.82075 10.7802 176.303 5290 1 seldon-batch-job-sdep-3 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER grpc 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 true
3 5.50125 5.18707 7.05242 7.99637 10.917 175.338 5261 1 seldon-batch-job-sdep-2 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER grpc 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false

@axsaucedo
Copy link
Contributor Author

/test benchmark

@axsaucedo
Copy link
Contributor Author

/test benchmark

@axsaucedo
Copy link
Contributor Author

/test benchmark

2 similar comments
@axsaucedo
Copy link
Contributor Author

/test benchmark

@axsaucedo
Copy link
Contributor Author

/test benchmark

@axsaucedo
Copy link
Contributor Author

/test benchmark

@axsaucedo
Copy link
Contributor Author

/test benchmark

@seldondev
Copy link
Collaborator

Benchmark results

mean 50th 90th 95th 99th throughputAchieved success errors name replicas serverWorkers serverThreads modelUri image server apiType requestsCpu requestsMemory limitsCpu limitsMemory benchmarkCpu concurrency duration rate disableOrchestrator
0 4.62294 4.48054 5.50417 5.95148 7.40162 216.218 6488 0 seldon-batch-job-sdep-1 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER rest 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 true
1 3.54699 3.38231 4.34344 4.85549 6.61593 271.79 8156 1 seldon-batch-job-sdep-2 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER grpc 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
2 3.55453 3.39792 4.34801 4.81771 6.4163 271.367 8143 1 seldon-batch-job-sdep-3 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER grpc 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 true
3 4.63803 4.49508 5.49917 5.91541 7.4068 215.52 6467 0 seldon-batch-job-sdep-0 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER rest 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false

@seldondev seldondev merged commit 48b71b9 into SeldonIO:master Jun 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants