Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating Argo Workflows link since repo renamed #3387

Merged
merged 1 commit into from
Jul 8, 2021

Conversation

axsaucedo
Copy link
Contributor

What this PR does / why we need it:

Argo Workflows has renamed their repo from argoproj/argo to argoproj/argo-workflows which has broken notebooks / tests so upgrading to latest.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

@axsaucedo
Copy link
Contributor Author

/test benchmark

@seldondev
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign axsaucedo
You can assign the PR to them by writing /assign @axsaucedo in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@axsaucedo
Copy link
Contributor Author

Argo workflows now fixed - merging

@axsaucedo axsaucedo merged commit 9bb1cfc into SeldonIO:master Jul 8, 2021
@seldondev
Copy link
Collaborator

Benchmark results - Testing Service Orchestrator

  • Orch added mean latency under 4ms: True
  • Orch added 95th latency under 5ms: True
  • Orch added 99th latency under 10ms: True
  • No errors: True

Results table

mean 50th 90th 95th 99th throughputAchieved success errors iteration_name replicas serverWorkers serverThreads modelUri image server apiType requestsCpu requestsMemory limitsCpu limitsMemory benchmarkCpu concurrency duration rate disableOrchestrator
3 3.18445 3.05391 3.90518 4.32066 5.96243 310.847 9328 1 seldon-benchmark-sdep-2 1 5 1 seldonio/seldontest_predict:1.10.0-dev grpc 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
2 2.67551 2.53196 3.46454 3.86922 5.15468 369.15 11078 1 seldon-benchmark-sdep-3 1 5 1 seldonio/seldontest_predict:1.10.0-dev grpc 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 true
1 4.40939 4.23786 5.16579 5.60023 7.21559 226.699 6802 0 seldon-benchmark-sdep-0 1 5 1 seldonio/seldontest_predict:1.10.0-dev rest 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
0 3.25967 3.15018 4.08006 4.45353 5.53453 306.603 9200 0 seldon-benchmark-sdep-1 1 5 1 seldonio/seldontest_predict:1.10.0-dev rest 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 true

@seldondev
Copy link
Collaborator

Benchmark Results - Python Wrapper V1 vs V2

  • V1 base mean performance latency under 10ms: True
  • V1 base 99th performance latenc under 10ms: True
  • V1 base throughput above 180rps: True
  • V1 base throughput above 250rps: True
  • V2 mean performance latency under 5ms: True
  • V2 99th performance latenc under 10ms: True
  • V2 REST throughput above 250rps: True
  • V2 throughput above 300rps: True

Python V1 Wrapper Results table

mean 50th 90th 95th 99th throughputAchieved success errors iteration_name replicas serverWorkers serverThreads modelUri image server apiType requestsCpu requestsMemory limitsCpu limitsMemory benchmarkCpu concurrency duration rate disableOrchestrator
2 4.57791 3.5523 6.60819 9.77274 23.636 216.08 6485 1 seldon-benchmark-sdep-3 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER grpc 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
4 5.24413 4.66073 7.37715 8.60601 12.1551 190.591 5720 0 seldon-benchmark-sdep-0 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER rest 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
3 343.998 342.115 370.497 387.246 417.729 435.279 12931 147 seldon-benchmark-sdep-5 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER grpc 2000Mi 500Mi 2000Mi 500Mi 1 150 30s 0 false
5 306.457 298.656 416.306 493.354 592.082 485.158 14637 0 seldon-benchmark-sdep-2 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER rest 2000Mi 500Mi 2000Mi 500Mi 1 150 30s 0 false
1 115.374 114.296 126.78 131.834 138.946 432.817 12944 49 seldon-benchmark-sdep-4 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER grpc 2000Mi 500Mi 2000Mi 500Mi 1 50 30s 0 false
0 113.988 100.892 194.144 198.954 206.911 432.623 12985 0 seldon-benchmark-sdep-1 1 5 1 gs://seldon-models/sklearn/iris SKLEARN_SERVER rest 2000Mi 500Mi 2000Mi 500Mi 1 50 30s 0 false

Python V2 MLServer Results table

mean 50th 90th 95th 99th throughputAchieved success errors iteration_name replicas serverWorkers serverThreads modelUri image server apiType requestsCpu requestsMemory limitsCpu limitsMemory benchmarkCpu concurrency duration rate disableOrchestrator
3 2.34518 2.15366 2.8806 3.26487 5.29824 420.119 12608 1 seldon-benchmark-sdep-3 1 5 1 gs://seldon-models/sklearn/iris-0.23.2/lr_model SKLEARN_SERVER grpc 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
1 30000.7 30000.7 30000.9 30000.9 30000.9 0 0 2 seldon-benchmark-sdep-0 1 5 1 gs://seldon-models/sklearn/iris-0.23.2/lr_model SKLEARN_SERVER rest 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
5 160.709 186.314 207.07 213.754 276.286 925.508 27652 150 seldon-benchmark-sdep-5 1 5 1 gs://seldon-models/sklearn/iris-0.23.2/lr_model SKLEARN_SERVER grpc 2000Mi 500Mi 2000Mi 500Mi 1 150 30s 0 false
2 301.032 296.416 446.545 502.718 609.308 494.005 14931 0 seldon-benchmark-sdep-2 1 5 1 gs://seldon-models/sklearn/iris-0.23.2/lr_model SKLEARN_SERVER rest 2000Mi 500Mi 2000Mi 500Mi 1 150 30s 0 false
4 64.7138 82.4356 100.389 103.084 110.193 769.173 23039 50 seldon-benchmark-sdep-4 1 5 1 gs://seldon-models/sklearn/iris-0.23.2/lr_model SKLEARN_SERVER grpc 2000Mi 500Mi 2000Mi 500Mi 1 50 30s 0 false
0 109.771 100.477 193.174 197.673 205.273 451.509 13562 0 seldon-benchmark-sdep-1 1 5 1 gs://seldon-models/sklearn/iris-0.23.2/lr_model SKLEARN_SERVER rest 2000Mi 500Mi 2000Mi 500Mi 1 50 30s 0 false

@seldondev
Copy link
Collaborator

Benchmark results - Testing Seldon V1 Data Types

Results for NDArray

mean 50th 90th 95th 99th throughputAchieved success errors iteration_name replicas serverWorkers serverThreads modelUri image server apiType requestsCpu requestsMemory limitsCpu limitsMemory benchmarkCpu concurrency duration rate disableOrchestrator
1 62.9642 60.2042 82.3176 86.5745 100.333 15.8539 475 1 seldon-benchmark-sdep-3 1 5 1 seldonio/seldontest_predict:1.10.0-dev grpc 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
2 8.87718 7.71665 12.9277 14.7399 19.5295 112.615 3380 0 seldon-benchmark-sdep-0 1 5 1 seldonio/seldontest_predict:1.10.0-dev rest 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
3 5039.94 5406.75 8726.93 9323.96 10916.9 29.6108 739 150 seldon-benchmark-sdep-5 1 5 1 seldonio/seldontest_predict:1.10.0-dev grpc 2000Mi 500Mi 2000Mi 500Mi 1 150 30s 0 false
0 450.22 413.378 694.913 779.831 956.549 327.559 9922 0 seldon-benchmark-sdep-2 1 5 1 seldonio/seldontest_predict:1.10.0-dev rest 2000Mi 500Mi 2000Mi 500Mi 1 150 30s 0 false
4 15000.2 nan nan nan nan 3.3315 0 100 seldon-benchmark-sdep-4 1 5 1 seldonio/seldontest_predict:1.10.0-dev grpc 2000Mi 500Mi 2000Mi 500Mi 1 50 30s 0 false
5 162.941 185.659 204.024 213.312 294.383 300.275 9028 0 seldon-benchmark-sdep-1 1 5 1 seldonio/seldontest_predict:1.10.0-dev rest 2000Mi 500Mi 2000Mi 500Mi 1 50 30s 0 false

Results for Tensor

mean 50th 90th 95th 99th throughputAchieved success errors iteration_name replicas serverWorkers serverThreads modelUri image server apiType requestsCpu requestsMemory limitsCpu limitsMemory benchmarkCpu concurrency duration rate disableOrchestrator
4 5.9524 4.47711 10.143 13.8614 23.2864 166.337 4991 1 seldon-benchmark-sdep-3 1 5 1 seldonio/seldontest_predict:1.10.0-dev grpc 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
1 8.78506 7.66484 12.6071 14.9755 19.8983 113.799 3415 0 seldon-benchmark-sdep-0 1 5 1 seldonio/seldontest_predict:1.10.0-dev rest 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
3 375.714 370.876 603.623 684.41 817.325 395.081 11750 142 seldon-benchmark-sdep-5 1 5 1 seldonio/seldontest_predict:1.10.0-dev grpc 2000Mi 500Mi 2000Mi 500Mi 1 150 30s 0 false
5 452.044 411.679 693.275 786.245 973.816 327.147 9911 0 seldon-benchmark-sdep-2 1 5 1 seldonio/seldontest_predict:1.10.0-dev rest 2000Mi 500Mi 2000Mi 500Mi 1 150 30s 0 false
0 151.466 152.856 208.744 224.447 295.857 327.206 9777 50 seldon-benchmark-sdep-4 1 5 1 seldonio/seldontest_predict:1.10.0-dev grpc 2000Mi 500Mi 2000Mi 500Mi 1 50 30s 0 false
2 163.008 185.707 204.253 210.893 293.484 300.614 9026 0 seldon-benchmark-sdep-1 1 5 1 seldonio/seldontest_predict:1.10.0-dev rest 2000Mi 500Mi 2000Mi 500Mi 1 50 30s 0 false

Results for TFTensor

mean 50th 90th 95th 99th throughputAchieved success errors iteration_name replicas serverWorkers serverThreads modelUri image server apiType requestsCpu requestsMemory limitsCpu limitsMemory benchmarkCpu concurrency duration rate disableOrchestrator
2 5.39485 4.21481 9.12962 11.7738 18.0394 183.407 5504 1 seldon-benchmark-sdep-3 1 5 1 seldonio/seldontest_predict:1.10.0-dev grpc 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
0 7.99733 7.03457 11.3255 13.47 18.3672 124.865 3749 0 seldon-benchmark-sdep-0 1 5 1 seldonio/seldontest_predict:1.10.0-dev rest 2000Mi 500Mi 2000Mi 500Mi 1 1 30s 0 false
5 342.858 326.79 403.233 416.098 513.751 435.763 12978 114 seldon-benchmark-sdep-5 1 5 1 seldonio/seldontest_predict:1.10.0-dev grpc 2000Mi 500Mi 2000Mi 500Mi 1 150 30s 0 false
3 462.676 431.3 704.118 804.398 989.142 318.7 9689 0 seldon-benchmark-sdep-2 1 5 1 seldonio/seldontest_predict:1.10.0-dev rest 2000Mi 500Mi 2000Mi 500Mi 1 150 30s 0 false
4 116.759 106.466 177.559 190.517 209.328 425.598 12729 50 seldon-benchmark-sdep-4 1 5 1 seldonio/seldontest_predict:1.10.0-dev grpc 2000Mi 500Mi 2000Mi 500Mi 1 50 30s 0 false
1 168.183 188.779 206.271 259.016 297.046 290.724 8752 0 seldon-benchmark-sdep-1 1 5 1 seldonio/seldontest_predict:1.10.0-dev rest 2000Mi 500Mi 2000Mi 500Mi 1 50 30s 0 false

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants