Adding Kuberenetes templates and deployment guide #1962

Langhalsdino · 2020-07-29T23:16:14Z

Motivation and context

The topic was raised a couple of times in issues like #1087 . Since kubernetes is widely use easy deployment into the kubernetes environment would provide great value to the community and help to get cvat to a wider audience.

Special due to changes like #1641 its now way easier to deploy cvat in a k8s environment.

How has this been tested?

I deployed this in a couple of namespaces with in our cluster (with and without nvida gpu).
Furthermore i did not do any changes to the code, therefore the only real issue was networking. Since i was following the docker-compose.yml closely there where no real challges

Checklist

I submit my changes into the develop branch
I have added description of my changes into CHANGELOG file
I have updated the documentation accordingly
I have added tests to cover my changes
I have linked related issues (read github docs)
I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below)

# Copyright (C) 2020 Intel Corporation
#
# SPDX-License-Identifier: MIT

coveralls · 2020-07-29T23:39:41Z

Pull Request Test Coverage Report for Build 7348

0 of 0 changed or added relevant lines in 0 files are covered.
1092 unchanged lines in 43 files lost coverage.
Overall coverage increased (+1.7%) to 70.78%

Files with Coverage Reduction	New Missed Lines	%
cvat/apps/authentication/urls.py	1	82.35%
datumaro/datumaro/components/launcher.py	1	80.36%
datumaro/datumaro/plugins/accuracy_checker_plugin/details/ac.py	1	4.17%
datumaro/datumaro/plugins/accuracy_checker_plugin/launcher.py	1	25.0%
datumaro/datumaro/plugins/voc_format/format.py	1	97.71%
datumaro/datumaro/util/init.py	1	79.59%
cvat/apps/authentication/auth_basic.py	3	75.0%
src/frames.js	3	23.31%
datumaro/datumaro/plugins/openvino_launcher.py	4	3.95%
src/labels.js	4	82.86%

Totals
Change from base Build 6672:	1.7%
Covered Lines:	12498
Relevant Lines:	17262

💛 - Coveralls

ActiveChooN · 2020-08-03T12:17:45Z

Hi @Langhalsdino, thanks a lot for your contribution. In our team we think a lot about adding kubernetes or docker swarm support for deployment. I have a few questions and suggestions.

We recently merged PR (DL models as serverless functions #1767) with DL models serving as a function with Nuclio. Do you want to add this in your templates?
There is a nice way to organize the kubernetes structure with Helm charts and helmfile. This can help to separate configuration values from templates itself and provides easy way to deploy and manage application. What about to use this tools in your PR?

Langhalsdino · 2020-08-03T15:02:55Z

There is a nice way to organize the kubernetes structure with Helm charts and helmfile. This can help to separate configuration values from templates itself and provides easy way to deploy and manage application. What about to use this tools in your PR?

Building helm files is in general a good idea and would help to set all the variable such as docker image registry, domains, ...
I will focus on this next weekend and try to add it with some new commits.

We recently merged PR (#1767) with DL models serving as a function with Nuclio. Do you want to add this in your templates?

I could add it to the template. I could not find any examples on how to use nuclio, do you have example models, ... such that i could test the deployment.

Lastly do you know the current status of #7 , since referencing official images in the deployment templates makes it way clearer and could prepare the ground for publishing cvat as an official helm chart on helm.sh

In general i am not a big fan of having growing pull requests. Therefore i would prefere if we could agree on a certain scope of this issue or maybe split it into multiple once, such that we will see them merged :)

nmanovic · 2020-08-03T17:01:24Z

@Langhalsdino , a couple of examples with nuclio you can find here: https://github.com/opencv/cvat/blob/develop/serverless/deploy.sh

What is the reasonable scope for the PR from your point of view?

Langhalsdino · 2020-08-03T20:44:30Z

What is the reasonable scope for the PR from your point of view?

I am tempted to split this into two issues and PR.
The first one is adding a helm chart for the basic setup of CVAT and second one would be extending it with a nuclio container (incl. GPU support).

Basic helm chart would include a statefull set of:

PostgresDB
Redis
CVAT
CVAT-UI
Service

The other issue would extend the functionality by adding nuclio support and GPU support for CVAT and Nuclio.

What do you think about it @nmanovic

rushtehrani · 2020-08-03T23:02:36Z

If the plan is to use Helm, note that Nuclio supports and recommends the Helm chart deployment for production.

GPU support is dependent on the provider, for example GKE automatically installs the NVIDIA device plugin daemonset but AKS and EKS require you to install it manually.

Also, now that inference is in Nuclio, do we still need to have GPU support in the main CVAT container?

Langhalsdino · 2020-08-03T23:09:16Z

GPU support is dependent on the provider, for example GKE automatically installs the NVIDIA device plugin daemonset but AKS and EKS require you to install it manually.

I know about the plugin deamonset and i would mention it in a Readme, such that every EKS and AKS user will read it after there pods are not being scheduled :D

If the plan is to use Helm, note that Nuclio supports and recommends the Helm chart deployment for production.

Thats looks awesome. Then we will not need to split this into two issues and i will just link to their helm repo and add configuration options for connecting to Nuclio.

Also, now that inference is in Nuclio, do we still need to have GPU support in the main CVAT container?

This will probably make it a lot easier. If the roadmap of CVAT is pursuing a deeper integration of nuclio, i am happy to skipp the GPU part. Since it is different for every platform 🙈

rushtehrani · 2020-08-03T23:17:16Z

That all sounds great! At some point we may also want to support a Kustomize deployment too.

Langhalsdino · 2020-08-03T23:19:39Z

That all sounds great! At some point we may also want to support a Kustomize deployment too.

Why do you want to go for Kustomize, if there is already a helm chart? I can not see the benefits of having both.
If kustomize ist the way to go, i could take a look into it and build it instead of helm. Eventhough helm is to be very popular and i use it on a daily bases.

rushtehrani · 2020-08-03T23:30:43Z

Why do you want to go for Kustomize, if there is already a helm chart? I can not see the benefits of having both.
If kustomize ist the way to go, i could take a look into it and build it instead of helm. Eventhough helm is to be very popular and i use it on a daily bases.

To clarify, I think going with Helm for this phase makes perfect sense.

Ideally we'd want to support both. Kustomize is built into kubectl and projects like ours and Kubeflow use it for mainly the reasons outlined in this overview.

We can contribute the Kustomize version at a later time.

nmanovic · 2020-08-04T07:06:31Z

What is the reasonable scope for the PR from your point of view?

I am tempted to split this into two issues and PR.
The first one is adding a helm chart for the basic setup of CVAT and second one would be extending it with a nuclio container (incl. GPU support).

Basic helm chart would include a statefull set of:

PostgresDB

Redis

CVAT

CVAT-UI

Service

The other issue would extend the functionality by adding nuclio support and GPU support for CVAT and Nuclio.

What do you think about it @nmanovic

Looks great! Thanks for the contribution.

kubernetes-templates/README.md

nmanovic · 2020-08-26T14:12:23Z

@Langhalsdino , should we review the PR again or it is in progress? Could you please fix codacy issues?

Langhalsdino · 2020-08-31T09:48:57Z

Sorry i got lost in over engineering a helm chart and interrupted by upcoming exams and my thesis. I will fix codacy issues this weekend and open another PR with the helm chart later on.

Langhalsdino · 2020-09-07T22:52:14Z

@nmanovic I fixed the the issues mentioned in codacy. Regarding the CHANGE.log file. Should i add the changes in the following section or is this done by the repo maintainers?

## [1.1.0-alpha] - 2020-06-30
### Added

nmanovic · 2020-10-03T03:55:52Z

@azhavoro , could you please look at the PR? Do you have any comments?

nmanovic · 2020-10-03T03:56:47Z

@Langhalsdino , sorry for the delay with our review and thanks for the contribution!

jizg · 2020-10-12T13:57:19Z

Hi, guys. I have tried deploying nuclio on my Minikube, and tested with CVAT deployed by @Langhalsdino 's YAML templates. The automatic annotation worked well in this way. I made some changes to the YAML templates, just created a service for nuclio in cvat namespace and pointed to nuclio-dashboard service in nuclio namespace, and added no_proxy env for nuclio service to bypass cvat_proxy in cvat_backend_deployment.yml.

Langhalsdino · 2020-10-12T18:44:57Z

Hi, guys. I have tried deploying nuclio on my Minikube, and tested with CVAT deployed by @Langhalsdino 's YAML templates. The automatic annotation worked well in this way. I made some changes to the YAML templates, just created a service for nuclio in cvat namespace and pointed to nuclio-dashboard service in nuclio namespace, and added no_proxy env for nuclio service to bypass cvat_proxy in cvat_backend_deployment.yml.

Could you share you changes, since i would like to include them in my template :)

jizg · 2020-10-13T07:25:23Z

Sure. Please check this PR apic-ai#1
nuclio can be deployed based on the offical guide here (https://github.com/nuclio/nuclio/tree/development/docs/setup).

nmanovic · 2020-10-14T11:07:06Z

@azhavoro , could you please work together with @Langhalsdino to merge the PR? I see that CI is failed. But the functionality is really important for our internal infrastructure. Take care about it.

Langhalsdino · 2020-10-20T10:47:28Z

Sure. Please check this PR apic-ai#1
nuclio can be deployed based on the offical guide here (https://github.com/nuclio/nuclio/tree/development/docs/setup).

@jizg thank you for the PR i merged it :)

fg91 · 2020-11-11T15:50:28Z

Hey @Langhalsdino ,

thanks for the kubernetes templates, looking really forward to this functionality!

When trying, I find that the cvat backend cannot find redis.

Backend logs:

2020-11-11 15:45:14,309 DEBG 'rqscheduler' stderr output:
15:45:14 Registering birth

2020-11-11 15:45:14,339 DEBG 'rqscheduler' stderr output:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/redis/connection.py", line 559, in connect
    sock = self._connect()
  File "/usr/local/lib/python3.8/dist-packages/redis/connection.py", line 584, in _connect
    for res in socket.getaddrinfo(self.host, self.port, self.socket_type,
  File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/rqscheduler", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/rq_scheduler/scripts/rqscheduler.py", line 61, in main
    scheduler.run(burst=args.burst)
  File "/usr/local/lib/python3.8/dist-packages/rq_scheduler/scheduler.py", line 435, in run
    self.register_birth()
  File "/usr/local/lib/python3.8/dist-packages/rq_scheduler/scheduler.py", line 60, in register_birth
    if self.connection.exists(self.key) and \
  File "/usr/local/lib/python3.8/dist-packages/redis/client.py", line 1581, in exists
    return self.execute_command('EXISTS', *names)
  File "/usr/local/lib/python3.8/dist-packages/redis/client.py", line 898, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/usr/local/lib/python3.8/dist-packages/redis/connection.py", line 1192, in get_connection
    connection.connect()
  File "/usr/local/lib/python3.8/dist-packages/redis/connection.py", line 563, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error -2 connecting to redis:6379. Name or service not known.

2020-11-11 15:45:14,355 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139846789984064 for <Subprocess at 139846790140640 with name rqscheduler in state STARTING> (stdout)>
2020-11-11 15:45:14,355 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 139846789737680 for <Subprocess at 139846790140640 with name rqscheduler in state STARTING> (stderr)>
2020-11-11 15:45:14,355 INFO exited: rqscheduler (exit status 1; not expected)
2020-11-11 15:45:14,355 DEBG received SIGCHLD indicating a child quit
2020-11-11 15:45:15,357 INFO gave up: rqscheduler entered FATAL state, too many start retries too quickly

Do you know why this could be happening?
Thanks!

Update:
The issue might be the following:

Here, --host redis is hardcoded.
https://github.com/openvinotoolkit/cvat/blob/b565721d81cd2357b82dd7cb9273e8eaf3b97222/supervisord.conf#L46

And in the docker-compose.yml a corresponding alias is specified:

  cvat_redis:
    container_name: cvat_redis
    image: redis:4.0-alpine
    networks:
      default:
        aliases:
          - redis

In kubernetes, the service is, however, not reachable under redis.

Langhalsdino · 2020-11-13T13:51:18Z

@nmanovic I am unable to view the snyk security output and therefore am not sure what failed. Could you share the output with me?

Langhalsdino · 2020-11-13T13:57:07Z

Hi @fg91 ,

since you send me an email regarding a different image i am assuming that you resolved the redis issue.
If so, could you share what resolved the issue. If not i am happy to take a look into it.

In your email you raised the issue that hosting cvat on a specific path of a domain likemy.cool.domain/cvat is not working. I have not tried this yet, but i would assume, that is currently not possible with cvat. Maybe a CVAT core developer can confirm or denial this.

If it is possible by cvat, i would investigate it further since it should be easily supported by adjusting the templates at a few places.

Langhalsdino · 2021-02-28T18:24:06Z

I am really sorry didn't responding to your request. Based on your previous messages i changed a couple of things:

Merged the current develop branch into the PR.
Fixed the kubernetes API, such that the latest kubernetes version v1.19.3 is supported and tested it for v1.19.3 and 1.9
Added changelog entry.

This should cover the above mentioned discussions.

coveralls · 2021-02-28T19:18:34Z

Coverage remained the same at 72.22% when pulling c773419 on apic-ai:release-1.1.0 into ca97507 on openvinotoolkit:develop.

Langhalsdino · 2021-02-28T19:34:46Z

@azhavoro and @nmanovic could you do a new code review of this PR? I am really sorry I kept you waiting.

nmanovic · 2021-03-01T18:40:45Z

@azhavoro , need to comment or merge ASAP.

Langhalsdino · 2021-03-02T11:23:33Z

@azhavoro If you need some assistance to setup a local dev environment or other questions, i am happy to assist.

kubernetes-templates/04_cvat_backend_deployment.yml

kubernetes-templates/05_cvat_proxy_service.yml

kubernetes-templates/04_cvat_frontend_deployment.yml

azhavoro · 2021-03-03T12:58:19Z

@Langhalsdino Great job! I tested the PR and it works for me with minor changes (my comments above).
Thank you for the contribution!

turowicz · 2021-03-03T14:03:11Z

Does nuclio work with Kubernetes >= 1.19.x?

@Langhalsdino recent commit mentioned it should work with 1.19.x.

Langhalsdino · 2021-03-03T14:05:14Z

Does nuclio work with Kubernetes >= 1.19.x?

@Langhalsdino recent commit mentioned it should work with 1.19.x.

I tested the template with 1.9 and 1.19 and therefore deduce that everything between these versions should work. I will update the README.md to include the feedback by @azhavoro this evening.

Since nuclio works for v1.7 or later according to its documentation, there should not be any compatibility issues.

turowicz · 2021-03-03T14:16:33Z

@Langhalsdino I wasn't able to get nuclio to work with Kubernetes >1.19 because it does not support Containerd. How did you host model containers without nuclio trying to build them?

Langhalsdino · 2021-03-04T12:12:54Z

I wasn't able to get nuclio to work with Kubernetes >1.19 because it does not support Containerd. How did you host model containers without nuclio trying to build them?

Sorry, i did not phrase this correctly, therefore this misunderstanding is my fault.

I tested the deployment (part of this PR) with Kubernetes 1.19 and 1.9. Since CVAT works without nuclio this was fine for kubernetes 1.19. Since kubernetes change of using containered was around 1.18, i was suggesting to downgrade your cluster to 1.17, if you would like to use nuclio or wait for nuclio to have fixed this issue.

azhavoro · 2021-03-09T07:55:27Z

LGTM
@ActiveChooN could you please review as well?

tamademicheli · 2021-03-11T14:18:53Z

actually no, sorry i have the pr almost ready but no time for that. don't wait for me, when i will have time i will see if my pr is still valid.

…

On Mon, Feb 22, 2021 at 10:54 PM Matt Hogan ***@***.***> wrote: Hi, i just cloned your work. What you are doing is something different than a template. I used your truly valuable work to build a helm chart (working with k8s 16+ and the latest endpoints), which is working on cpu and without further binding to nuclio. I removed some part which were not necessary to be declared like redis, postgres and nuclio which i pull them as a chart dependency. The result is a chart with 2 deployments, frontend and backend. Is someone interested on us to publish a helm chart? if yes where would be the most suitable repo to do that ? I don't like code and chart on the same repo. I do have difficulties in applying the original yaml files, which needs many changes and currently still doesn't work for me. so if you can share the helm chart, that will be awesome! Hi I will try to create a PR this week. i have to write the README.md and add some script to install kind k8s sig with the ingress controllers. Any update on this? I had plans to leverage the manifests provided in this PR and create a Helm chart as well, but would love to re-use your submission. I am happy to assist with any missing elements if needed. All contributions are much appreciated! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1962 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF5ZGSLU7UERK44N6OKQ42TTALG7XANCNFSM4PMIYBIQ> .

ActiveChooN · 2021-03-17T08:19:31Z

@azhavoro, LGTM

nmanovic

@Langhalsdino , thanks for the great contribution!

ActiveChooN · 2021-04-28T17:51:22Z

@Langhalsdino, hi, we've just merged #3102 helm charts, which can easily replace raw k8s templates. Is there any special purpose for k8s templates or its can be removed?

jeffliao888 · 2022-07-13T02:15:09Z

I following the document and deploy image: openvino/cvat_server:v1.7.0 in my Kubernetes v1.23.8 , and it seems work fine, but when I changed the image: openvino/cvat_server:v2.x.0 , then it showing "Could not check authorization on the server" in the brower.

Can someone help me if you can success to upgrade the CVAT server image from v1.x.0 to v2.x.0 and work fine in K8s ?
Thank you !

Langhalsdino requested a review from nmanovic as a code owner July 29, 2020 23:16

aaniin reviewed Aug 18, 2020

View reviewed changes

kubernetes-templates/README.md Show resolved Hide resolved

Langhalsdino mentioned this pull request Oct 1, 2020

Add more environment variables to configure CVAT container #445

Closed

nmanovic requested a review from azhavoro October 3, 2020 04:01

Langhalsdino closed this Oct 12, 2020

Langhalsdino reopened this Oct 12, 2020

addd basic cvat deployment k8s templates

a29c880

Langhalsdino force-pushed the release-1.1.0 branch from 7407c68 to a29c880 Compare February 28, 2021 18:19

Langhalsdino added 3 commits February 28, 2021 19:26

lined README.md

fefc2fb

switched to prebuild images

24b65c2

added fixed image version, added further debugging hints

726e352

lined readme

6766a2c

azhavoro reviewed Mar 3, 2021

View reviewed changes

kubernetes-templates/04_cvat_backend_deployment.yml Outdated Show resolved Hide resolved

kubernetes-templates/05_cvat_proxy_service.yml Show resolved Hide resolved

kubernetes-templates/04_cvat_frontend_deployment.yml Outdated Show resolved Hide resolved

Langhalsdino added 2 commits March 4, 2021 10:50

removed deprecated envirorment variables in the fronend deployment

590d7f0

added guide on how to expose the cluster to readme

3446732

Merge branch 'develop' into release-1.1.0

c773419

azhavoro approved these changes Mar 17, 2021

View reviewed changes

nmanovic approved these changes Mar 18, 2021

View reviewed changes

nmanovic merged commit 54ee8a1 into cvat-ai:develop Mar 18, 2021

Langhalsdino deleted the release-1.1.0 branch March 18, 2021 17:38

ActiveChooN mentioned this pull request May 7, 2021

Remove raw k8s templates #3171

Merged

5 tasks

Adding Kuberenetes templates and deployment guide #1962

Adding Kuberenetes templates and deployment guide #1962

Conversation

Langhalsdino commented Jul 29, 2020 • edited Loading

Motivation and context

How has this been tested?

Checklist

License

coveralls commented Jul 29, 2020 • edited Loading

Pull Request Test Coverage Report for Build 7348

💛 - Coveralls

ActiveChooN commented Aug 3, 2020

Langhalsdino commented Aug 3, 2020 • edited Loading

nmanovic commented Aug 3, 2020

Langhalsdino commented Aug 3, 2020 • edited Loading

rushtehrani commented Aug 3, 2020 • edited Loading

Langhalsdino commented Aug 3, 2020 • edited Loading

rushtehrani commented Aug 3, 2020

Langhalsdino commented Aug 3, 2020 • edited Loading

rushtehrani commented Aug 3, 2020

nmanovic commented Aug 4, 2020

nmanovic commented Aug 26, 2020

Langhalsdino commented Aug 31, 2020

Langhalsdino commented Sep 7, 2020

nmanovic commented Oct 3, 2020

nmanovic commented Oct 3, 2020

jizg commented Oct 12, 2020

Langhalsdino commented Oct 12, 2020

jizg commented Oct 13, 2020

nmanovic commented Oct 14, 2020

Langhalsdino commented Oct 20, 2020

fg91 commented Nov 11, 2020 • edited Loading

Langhalsdino commented Nov 13, 2020

Langhalsdino commented Nov 13, 2020 • edited Loading

Langhalsdino commented Feb 28, 2021

coveralls commented Feb 28, 2021 • edited Loading

Langhalsdino commented Feb 28, 2021

nmanovic commented Mar 1, 2021

Langhalsdino commented Mar 2, 2021

azhavoro commented Mar 3, 2021

turowicz commented Mar 3, 2021 • edited Loading

Langhalsdino commented Mar 3, 2021 • edited Loading

turowicz commented Mar 3, 2021 • edited Loading

Langhalsdino commented Mar 4, 2021

azhavoro commented Mar 9, 2021

tamademicheli commented Mar 11, 2021 via email

ActiveChooN commented Mar 17, 2021

nmanovic left a comment

Choose a reason for hiding this comment

ActiveChooN commented Apr 28, 2021

jeffliao888 commented Jul 13, 2022 • edited Loading

Langhalsdino commented Jul 29, 2020 •

edited

Loading

coveralls commented Jul 29, 2020 •

edited

Loading

Langhalsdino commented Aug 3, 2020 •

edited

Loading

Langhalsdino commented Aug 3, 2020 •

edited

Loading

rushtehrani commented Aug 3, 2020 •

edited

Loading

Langhalsdino commented Aug 3, 2020 •

edited

Loading

Langhalsdino commented Aug 3, 2020 •

edited

Loading

fg91 commented Nov 11, 2020 •

edited

Loading

Langhalsdino commented Nov 13, 2020 •

edited

Loading

coveralls commented Feb 28, 2021 •

edited

Loading

turowicz commented Mar 3, 2021 •

edited

Loading

Langhalsdino commented Mar 3, 2021 •

edited

Loading

turowicz commented Mar 3, 2021 •

edited

Loading

jeffliao888 commented Jul 13, 2022 •

edited

Loading