Added SiamMask CUDA implementation [2x faster on Full HD] #3571

bsekachev · 2021-08-19T09:14:59Z

General

Added CUDA support for SiamMask tracker
Tracker is able to track different objects in one server request
"Trackable" objects are introduced (an object will be tracked automatically if the previous frame was a latest keyframe for the object)
Added tracker indication on canvas
Added tracker switcher on sidebar
Tracking with non-blocking UI

Motivation and context

Resolved #2949
Resolved #2773
Related #2936

How has this been tested?

Checklist

I submit my changes into the develop branch
I have added description of my changes into CHANGELOG file
I have updated the documentation accordingly
I have added tests to cover my changes
I have linked related issues (read github docs)
I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below)

# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

bsekachev · 2021-08-19T09:16:39Z

serverless/pytorch/foolwood/siammask/nuclio/function.yaml

@@ -18,10 +18,20 @@ spec:

  build:
    image: cvat/pth.foolwood.siammask
-    baseImage: continuumio/miniconda3
+    baseImage: ubuntu:20.04


Changed base image to have the same set of preCopy directives.
But finally it was nesessary to add
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
to function-gpu.yml, so it is not the same now anyway. Let me know if you think that better to leave this file as is.

rohitsaluja22 · 2021-09-02T13:42:44Z

hi @bsekachev i am new to cvat, request you explain how to run SiamMask CUDA implementation from starting?
Do I need to modify serverless/pytorch/foolwood/siammask/nuclio/function.yaml, which other files should i modify? I am unable to find function-gpu.yml in the code I downloaded from github.

bsekachev · 2021-09-02T14:37:57Z

Hi @rohitsaluja22

Probably you use develop branch. This PR is not merged yet. To get function-gpu.yml you need to checkout to branch bs/siammask_gpu. Please, refer to git documentation.

To deploy the function you either run serverless/deploy_gpu.sh (to deploy all the gpu functions), or rename function-gpu.yml to function.yml and run nuctl deploy --project-name cvat --path `pwd`/serverless/pytorch/foolwood/siammask/nuclio/ --volume `pwd`/serverless/common:/opt/nuclio/common --platform local from cvat root directory.

ActiveChooN

LGTM, but not able to test GPU version. And one question: is there option to autoupdate function with updating CVAT images?

ActiveChooN · 2021-09-07T06:32:14Z

Successfully built serverless function with GPU support under WSL, but eventually got an error when called a function. Probably it's a problem with my setup.

Nuclio function log

21.09.06 22:50:48.605 processor (I) Starting processor {"version": "Label: 1.6.3, Git commit: 20b43c6d5637434ff9732c7fa8ce28c70d3805ce, OS: linux, Arch: amd64, Go version: go1.14.3"}

21.09.06 22:50:48.605 processor (D) Read configuration {"config": "{\n \"metadata\": {\n \"name\": \"pth-foolwood-siammask\",\n \"namespace\": \"nuclio\",\n \"labels\": {\n \"nuclio.io/project-name\": \"cvat\"\n },\n \"annotations\": {\n \"framework\": \"pytorch\",\n \"name\": \"SiamMask\",\n \"spec\": \"\",\n \"type\": \"tracker\"\n }\n },\n \"spec\": {\n \"description\": \"Fast Online Object Tracking and Segmentation\",\n \"handler\": \"main:handler\",\n \"runtime\": \"python:3.6\",\n \"env\": [\n {\n \"name\": \"PYTHONPATH\",\n \"value\": \"/opt/nuclio/SiamMask:/opt/nuclio/SiamMask/experiments/siammask_sharp\"\n }\n ],\n \"resources\": {\n \"limits\": {\n \"nvidia.com/gpu\": \"1\"\n }\n },\n \"image\": \"cvat/pth.foolwood.siammask:latest\",\n \"targetCPU\": 75,\n \"triggers\": {\n \"myHttpTrigger\": {\n \"class\": \"\",\n \"kind\": \"http\",\n \"name\": \"myHttpTrigger\",\n \"maxWorkers\": 2,\n \"workerAvailabilityTimeoutMilliseconds\": 10000,\n \"attributes\": {\n \"maxRequestBodySize\": 33554432\n }\n }\n },\n \"volumes\": [\n {\n \"volume\": {\n \"name\": \"volume-1\",\n \"hostPath\": {\n \"path\": \"/home/achoon/wsl_projects/cvat/serverless/common\"\n }\n },\n \"volumeMount\": {\n \"name\": \"volume-1\",\n \"mountPath\": \"/opt/nuclio/common\"\n }\n }\n ],\n \"build\": {\n \"functionConfigPath\": \"/home/achoon/wsl_projects/cvat/serverless/pytorch/foolwood/siammask/nuclio/function-gpu.yaml\",\n \"image\": \"cvat/pth.foolwood.siammask\",\n \"baseImage\": \"nvidia/cuda:11.1-devel-ubuntu20.04\",\n \"directives\": {\n \"preCopy\": [\n {\n \"kind\": \"ENV\",\n \"value\": \"PATH=\\\"/root/miniconda3/bin:${PATH}\\\"\"\n },\n {\n \"kind\": \"ARG\",\n \"value\": \"PATH=\\\"/root/miniconda3/bin:${PATH}\\\"\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"apt update \\u0026\\u0026 apt install -y --no-install-recommends wget git ca-certificates libglib2.0-0 libsm6 libxrender1 libxext6 \\u0026\\u0026 rm -rf /var/lib/apt/lists/*\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh \\u0026\\u0026 chmod +x Miniconda3-latest-Linux-x86_64.sh \\u0026\\u0026 ./Miniconda3-latest-Linux-x86_64.sh -b \\u0026\\u0026 rm -f Miniconda3-latest-Linux-x86_64.sh\"\n },\n {\n \"kind\": \"WORKDIR\",\n \"value\": \"/opt/nuclio\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"conda create -y -n siammask python=3.7\"\n },\n {\n \"kind\": \"SHELL\",\n \"value\": \"[\\\"conda\\\", \\\"run\\\", \\\"-n\\\", \\\"siammask\\\", \\\"/bin/bash\\\", \\\"-c\\\"]\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"git clone https://github.com/foolwood/SiamMask.git\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"pip install -r SiamMask/requirements.txt jsonpickle\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"conda install -y gcc_linux-64\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"cd SiamMask \\u0026\\u0026 bash make.sh \\u0026\\u0026 cd -\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"wget -P SiamMask/experiments/siammask_sharp http://www.robots.ox.ac.uk/~qwang/SiamMask_DAVIS.pth\"\n },\n {\n \"kind\": \"ENTRYPOINT\",\n \"value\": \"[\\\"conda\\\", \\\"run\\\", \\\"-n\\\", \\\"siammask\\\"]\"\n }\n ]\n },\n \"codeEntryType\": \"image\",\n \"timestamp\": 1630967944\n },\n \"platform\": {\n \"attributes\": {\n \"mountMode\": \"volume\",\n \"restartPolicy\": {\n \"maximumRetryCount\": 3,\n \"name\": \"always\"\n }\n }\n },\n \"readinessTimeoutSeconds\": 60,\n \"securityContext\": {},\n \"eventTimeout\": \"30s\"\n },\n \"PlatformConfig\": null\n}", "platformConfig": "{\n \"kind\": \"local\",\n \"webAdmin\": {\n \"enabled\": true,\n \"listenAddress\": \":8081\"\n },\n \"healthCheck\": {\n \"enabled\": true,\n \"listenAddress\": \":8082\"\n },\n \"logger\": {\n \"sinks\": {\n \"stdout\": {\n \"kind\": \"stdout\"\n }\n },\n \"system\": [\n {\n \"level\": \"debug\",\n \"sink\": \"stdout\"\n }\n ],\n \"functions\": [\n {\n \"level\": \"debug\",\n \"sink\": \"stdout\"\n }\n ]\n },\n \"metrics\": {},\n \"scaleToZero\": {},\n \"autoScale\": {},\n \"cronTriggerCreationMode\": \"processor\",\n \"ingressConfig\": {},\n \"kube\": {\n \"defaultServiceType\": \"ClusterIP\"\n },\n \"local\": {\n \"FunctionContainersHealthinessEnabled\": false,\n \"FunctionContainersHealthinessTimeout\": 5000000000,\n \"FunctionContainersHealthinessInterval\": 30000000000\n },\n \"imageRegistryOverrides\": {}\n}"}

21.09.06 22:50:48.606 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}

21.09.06 22:50:48.606 processor.http (D) Creating worker pool {"num": 2}

21.09.06 22:50:48.607 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-c4r9mi0n0a9vutmvhrng.sock"}

21.09.06 22:50:48.607 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-c4r9mi0n0a9vutmvhro0.sock"}

21.09.06 22:50:48.607 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}

21.09.06 22:50:48.607 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}

21.09.06 22:50:48.607 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}

21.09.06 22:50:48.607 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}

21.09.06 22:50:48.607 sor.http.w1.python.logger (D) Using Python executable {"path": "/root/miniconda3/envs/siammask/bin/python3"}

21.09.06 22:50:48.607 sor.http.w0.python.logger (D) Using Python executable {"path": "/root/miniconda3/envs/siammask/bin/python3"}

21.09.06 22:50:48.607 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/SiamMask:/opt/nuclio/SiamMask/experiments/siammask_sharp"}

21.09.06 22:50:48.607 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/SiamMask:/opt/nuclio/SiamMask/experiments/siammask_sharp"}

21.09.06 22:50:48.607 sor.http.w1.python.logger (D) Running wrapper {"command": "/root/miniconda3/envs/siammask/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-c4r9mi0n0a9vutmvhrng.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger --decode-event-strings"}

21.09.06 22:50:48.607 sor.http.w0.python.logger (D) Running wrapper {"command": "/root/miniconda3/envs/siammask/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-c4r9mi0n0a9vutmvhro0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger --decode-event-strings"}

21.09.06 22:50:53.750 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1, "pid": 312}

21.09.06 22:50:53.750 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 313}

l{"datetime": "2021-09-06 22:50:53,750", "level": "info", "message": "Replacing logger output", "with": {"handler_name": "default", "worker_id": "0"}}

21.09.06 22:50:53.750 sor.http.w0.python.logger (D) Waiting for start

21.09.06 22:50:53.750 sor.http.w1.python.logger (D) Waiting for start

l{"datetime": "2021-09-06 22:50:53,750", "level": "info", "message": "Replacing logger output", "with": {"handler_name": "default", "worker_id": "1"}}

21.09.06 22:50:53.752 sor.http.w1.python.logger (I) Init context... 0% {"worker_id": "1"}

21.09.06 22:50:53.752 sor.http.w0.python.logger (I) Init context... 0% {"worker_id": "0"}

21.09.06 22:51:33.349 sor.http.w1.python.logger (W) Failed to read from connection {"err": "EOF"}

21.09.06 22:51:33.414 sor.http.w1.python.logger (E) Unexpected termination of child process {"error": null, "status": "signal: killed"}



ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['processor']' command failed. (See above for error)

[2021-09-06 22:39:16,152-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:39:16,154-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:39:16,159-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:39:16,162-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:39:16,259-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

[2021-09-06 22:39:16,264-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

[2021-09-06 22:39:26,795-rk0-load_helper.py# 25] remove prefix 'module.'

[2021-09-06 22:39:26,795-rk0-load_helper.py# 25] remove prefix 'module.'

[2021-09-06 22:39:26,802-rk0-load_helper.py# 18] used keys:356

[2021-09-06 22:39:26,802-rk0-load_helper.py# 18] used keys:356

panic: Wrapper process for worker 1 exited unexpectedly with: signal: killed


goroutine 36 [running]:

github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc0003826c0)

/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576

created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper

/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5

--- Logging error ---

Traceback (most recent call last):

File "/opt/nuclio/_nuclio_wrapper.py", line 107, in serve_requests

event_message_length = self._resolve_event_message_length()

File "/opt/nuclio/_nuclio_wrapper.py", line 213, in _resolve_event_message_length

raise WrapperFatalException('Client disconnected')

WrapperFatalException: Client disconnected


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1029, in emit

self.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1009, in flush

self.stream.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe

Call stack:

File "/opt/nuclio/_nuclio_wrapper.py", line 383, in <module>

run_wrapper()

File "/opt/nuclio/_nuclio_wrapper.py", line 377, in run_wrapper

wrapper_instance.serve_requests()

File "/opt/nuclio/_nuclio_wrapper.py", line 121, in serve_requests

self._on_serving_error(exc)

File "/opt/nuclio/_nuclio_wrapper.py", line 241, in _on_serving_error

self._log_and_response_error(exc, 'Exception caught while serving')

File "/opt/nuclio/_nuclio_wrapper.py", line 250, in _log_and_response_error

self._logger.error_with(error_message, exc=str(exc), traceback=traceback.format_exc())

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 107, in error_with

self._update_bound_vars_and_log(logging.ERROR, message, *args, **kw_args)

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 116, in _update_bound_vars_and_log

self._logger._log(level, message, args, extra={"with": kw_args})

Message: 'Exception caught while serving'

Arguments: ()


ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['processor']' command failed. (See above for error)

[2021-09-06 22:43:40,250-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:43:40,252-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:43:40,252-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:43:40,254-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:43:40,329-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

[2021-09-06 22:43:40,330-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

panic: Wrapper process for worker 0 exited unexpectedly with: signal: killed


goroutine 13 [running]:

github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00002e000)

/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576

created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper

/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5

[2021-09-06 22:43:56,864-rk0-load_helper.py# 25] remove prefix 'module.'

[2021-09-06 22:43:56,933-rk0-load_helper.py# 18] used keys:356

--- Logging error ---

Traceback (most recent call last):

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1029, in emit

self.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1009, in flush

self.stream.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe

Call stack:

File "/opt/nuclio/_nuclio_wrapper.py", line 383, in <module>

run_wrapper()

File "/opt/nuclio/_nuclio_wrapper.py", line 367, in run_wrapper

args.decode_event_strings)

File "/opt/nuclio/_nuclio_wrapper.py", line 91, in __init__

getattr(entrypoint_module, 'init_context')(self._context)

File "/opt/nuclio/main.py", line 14, in init_context

context.logger.info("Init context...100%")

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 89, in info

self._update_bound_vars_and_log(logging.INFO, message, *args)

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 116, in _update_bound_vars_and_log

self._logger._log(level, message, args, extra={"with": kw_args})

Message: 'Init context...100%'

Arguments: ()

--- Logging error ---

Traceback (most recent call last):

File "/opt/nuclio/_nuclio_wrapper.py", line 367, in run_wrapper

args.decode_event_strings)

File "/opt/nuclio/_nuclio_wrapper.py", line 97, in __init__

self._write_packet_to_processor('s')

File "/opt/nuclio/_nuclio_wrapper.py", line 202, in _write_packet_to_processor

self._processor_sock_wfile.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1029, in emit

self.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1009, in flush

self.stream.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe

Call stack:

File "/opt/nuclio/_nuclio_wrapper.py", line 383, in <module>

run_wrapper()

File "/opt/nuclio/_nuclio_wrapper.py", line 372, in run_wrapper

traceback=traceback.format_exc())

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 107, in error_with

self._update_bound_vars_and_log(logging.ERROR, message, *args, **kw_args)

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 116, in _update_bound_vars_and_log

self._logger._log(level, message, args, extra={"with": kw_args})

Message: 'Caught unhandled exception while initializing'

Arguments: ()


ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['processor']' command failed. (See above for error)

[2021-09-06 22:44:10,883-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:44:10,884-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:44:10,885-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:44:10,885-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:44:10,950-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

[2021-09-06 22:44:10,954-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

[2021-09-06 22:44:24,165-rk0-load_helper.py# 25] remove prefix 'module.'

[2021-09-06 22:44:24,168-rk0-load_helper.py# 25] remove prefix 'module.'

[2021-09-06 22:44:24,218-rk0-load_helper.py# 18] used keys:356

[2021-09-06 22:44:24,219-rk0-load_helper.py# 18] used keys:356

panic: Wrapper process for worker 1 exited unexpectedly with: signal: killed


goroutine 65 [running]:

github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc0003826c0)

/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576

created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper

/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5

--- Logging error ---

Traceback (most recent call last):

File "/opt/nuclio/_nuclio_wrapper.py", line 107, in serve_requests

event_message_length = self._resolve_event_message_length()

File "/opt/nuclio/_nuclio_wrapper.py", line 213, in _resolve_event_message_length

raise WrapperFatalException('Client disconnected')

WrapperFatalException: Client disconnected


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1029, in emit

self.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1009, in flush

self.stream.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe

Call stack:

File "/opt/nuclio/_nuclio_wrapper.py", line 383, in <module>

run_wrapper()

File "/opt/nuclio/_nuclio_wrapper.py", line 377, in run_wrapper

wrapper_instance.serve_requests()

File "/opt/nuclio/_nuclio_wrapper.py", line 121, in serve_requests

self._on_serving_error(exc)

File "/opt/nuclio/_nuclio_wrapper.py", line 241, in _on_serving_error

self._log_and_response_error(exc, 'Exception caught while serving')

File "/opt/nuclio/_nuclio_wrapper.py", line 250, in _log_and_response_error

self._logger.error_with(error_message, exc=str(exc), traceback=traceback.format_exc())

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 107, in error_with

self._update_bound_vars_and_log(logging.ERROR, message, *args, **kw_args)

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 116, in _update_bound_vars_and_log

self._logger._log(level, message, args, extra={"with": kw_args})

Message: 'Exception caught while serving'

Arguments: ()


ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['processor']' command failed. (See above for error)

[2021-09-06 22:50:55,085-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:50:55,088-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:50:55,119-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:50:55,122-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:50:55,243-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

[2021-09-06 22:50:55,267-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

panic: Wrapper process for worker 1 exited unexpectedly with: signal: killed


goroutine 42 [running]:

github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00035e6c0)

/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576

created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper

/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5

[2021-09-07 06:06:36,120-rk0-load_helper.py# 25] remove prefix 'module.'

[2021-09-07 06:06:36,566-rk0-load_helper.py# 18] used keys:356

--- Logging error ---

Traceback (most recent call last):

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1029, in emit

self.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1009, in flush

self.stream.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe

Call stack:

File "/opt/nuclio/_nuclio_wrapper.py", line 383, in <module>

run_wrapper()

File "/opt/nuclio/_nuclio_wrapper.py", line 367, in run_wrapper

args.decode_event_strings)

File "/opt/nuclio/_nuclio_wrapper.py", line 91, in __init__

getattr(entrypoint_module, 'init_context')(self._context)

File "/opt/nuclio/main.py", line 14, in init_context

context.logger.info("Init context...100%")

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 89, in info

self._update_bound_vars_and_log(logging.INFO, message, *args)

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 116, in _update_bound_vars_and_log

self._logger._log(level, message, args, extra={"with": kw_args})

Message: 'Init context...100%'

Arguments: ()

--- Logging error ---

Traceback (most recent call last):

File "/opt/nuclio/_nuclio_wrapper.py", line 367, in run_wrapper

args.decode_event_strings)

File "/opt/nuclio/_nuclio_wrapper.py", line 97, in __init__

self._write_packet_to_processor('s')

File "/opt/nuclio/_nuclio_wrapper.py", line 202, in _write_packet_to_processor

self._processor_sock_wfile.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1029, in emit

self.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1009, in flush

self.stream.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe

Call stack:

File "/opt/nuclio/_nuclio_wrapper.py", line 383, in <module>

run_wrapper()

File "/opt/nuclio/_nuclio_wrapper.py", line 372, in run_wrapper

traceback=traceback.format_exc())

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 107, in error_with

self._update_bound_vars_and_log(logging.ERROR, message, *args, **kw_args)

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 116, in _update_bound_vars_and_log

self._logger._log(level, message, args, extra={"with": kw_args})

Message: 'Caught unhandled exception while initializing'

Arguments: ()

ActiveChooN · 2021-09-07T06:40:13Z

@rohitsaluja22, also you can not rename a file and just add the flag --file to nuctl deploy command with the path to function-gpu.yml file.

nmanovic · 2021-09-24T13:44:54Z

@bsekachev , if annotations are removed using "Remove annotations", tracked objects always are reinitialized.

…ting & history

bsekachev · 2021-09-29T10:08:10Z

@aschernov

Andrey, could your team update tracking documentation?
Thanks

aschernov · 2021-09-29T11:08:41Z

@bsekachev, sure, we'll do it.

siddtmb · 2024-03-19T15:47:10Z

I dont know why automatic annotation for "Allow Siammask active tracker to run for more frames" was closed, at the moment it can only run on the frame you are on and is incredibly slow

bsekachev added 3 commits August 19, 2021 10:13

Reworked function.yaml

d3c4a37

Added function-gpu

90e637c

Changed python version

bb9fb6f

bsekachev requested a review from nmanovic August 19, 2021 09:14

bsekachev commented Aug 19, 2021

View reviewed changes

bsekachev added 8 commits August 24, 2021 13:11

Merge branch 'develop' into bs/siammask_gpu

4e88b3b

Initial version of the new tracking

bc03050

Blocking navigation during tracking

9f576f4

Removed outdated code

88db3f4

Added canvas indication for trackable objects

0db2904

Support different trackers

da705ac

Fixed styles

0bafb8f

Added ability to add/remove trackable objects

ccee79f

bsekachev changed the title ~~[WIP] Added SiamMask CUDA implementation [2x faster on Full HD]~~ Added SiamMask CUDA implementation [2x faster on Full HD] Aug 31, 2021

bsekachev and others added 7 commits August 31, 2021 10:17

Fixed minor bug with text on the canvas

d1ca051

Updated version

20c9f08

updated changelog

6ec23ee

Merge branch 'develop' into bs/siammask_gpu

e8b3afc

Fixed a couple of issues

ac6d1b7

Updated changelog

15d6345

Explicit enumerable property

0cb4bec

bsekachev assigned ActiveChooN Sep 1, 2021

bsekachev requested a review from ActiveChooN September 1, 2021 11:03

Merge branch 'develop' into bs/siammask_gpu

47c8645

ActiveChooN previously approved these changes Sep 6, 2021

View reviewed changes

ActiveChooN mentioned this pull request Sep 7, 2021

Added documentation for GPU support in WSL #3651

Merged

4 tasks

bsekachev added 2 commits September 21, 2021 09:40

Removed outdated code

f9211c5

Fixed navigation

c7c09ef

bsekachev dismissed ActiveChooN’s stale review via c7c09ef September 21, 2021 17:20

Boris Sekachev added 2 commits September 27, 2021 16:35

Fixed issue with removing annotations, fixing issue with initial moun…

3e7dc1f

…ting & history

Merged develop

9f574b6

nmanovic approved these changes Sep 29, 2021

View reviewed changes

nmanovic merged commit dbdcd4f into develop Sep 29, 2021

nmanovic deleted the bs/siammask_gpu branch September 29, 2021 09:31

afilippx mentioned this pull request Oct 1, 2021

Documentation update, added information about trackers #3754

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added SiamMask CUDA implementation [2x faster on Full HD] #3571

Added SiamMask CUDA implementation [2x faster on Full HD] #3571

bsekachev commented Aug 19, 2021 •

edited

Loading

bsekachev Aug 19, 2021

rohitsaluja22 commented Sep 2, 2021 •

edited

Loading

bsekachev commented Sep 2, 2021

ActiveChooN left a comment •

edited

Loading

ActiveChooN commented Sep 7, 2021

ActiveChooN commented Sep 7, 2021

nmanovic commented Sep 24, 2021

bsekachev commented Sep 29, 2021 •

edited

Loading

aschernov commented Sep 29, 2021

siddtmb commented Mar 19, 2024

Added SiamMask CUDA implementation [2x faster on Full HD] #3571

Added SiamMask CUDA implementation [2x faster on Full HD] #3571

Conversation

bsekachev commented Aug 19, 2021 • edited Loading

General

Motivation and context

How has this been tested?

Checklist

License

bsekachev Aug 19, 2021

Choose a reason for hiding this comment

rohitsaluja22 commented Sep 2, 2021 • edited Loading

bsekachev commented Sep 2, 2021

ActiveChooN left a comment • edited Loading

Choose a reason for hiding this comment

ActiveChooN commented Sep 7, 2021

ActiveChooN commented Sep 7, 2021

nmanovic commented Sep 24, 2021

bsekachev commented Sep 29, 2021 • edited Loading

aschernov commented Sep 29, 2021

siddtmb commented Mar 19, 2024

bsekachev commented Aug 19, 2021 •

edited

Loading

rohitsaluja22 commented Sep 2, 2021 •

edited

Loading

ActiveChooN left a comment •

edited

Loading

bsekachev commented Sep 29, 2021 •

edited

Loading