Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue loading openvino models for semi-automatic and automatic annotation #1977

Closed
creativesiva opened this issue Aug 3, 2020 · 22 comments · Fixed by #1988 or #1996
Closed

Issue loading openvino models for semi-automatic and automatic annotation #1977

creativesiva opened this issue Aug 3, 2020 · 22 comments · Fixed by #1988 or #1996
Assignees
@nmanovic
Labels
bug Something isn't working
Milestone
1.1.0-release

Comments

@creativesiva
Copy link

I am getting error while trying to deploy openvino and pytorch models to nuclio (serverless functions), I have used 'bash serverless/deploy.sh to deploy models. Below is the log error for semantic-segmentation-adas-0001

Error - Container wasn't healthy in time
/nuclio/pkg/dockerclient/shell.go:429

Call stack:
Container wasn't healthy in time
/nuclio/pkg/dockerclient/shell.go:429
Function wasn't ready in time. Logs:

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

/nuclio/pkg/platform/local/platform.go:628

As a end results, it is installing i am able to deploy tensorflow models, below is the error message for rest of the models.
NAMESPACE | NAME | PROJECT | STATE | NODE PORT | REPLICAS
nuclio | openvino.dextr | cvat | error | 0 | 1/1
nuclio | openvino.omz.intel.person-reidentification-retail-0300 | cvat | error | 0 | 1/1
nuclio | openvino.omz.intel.text-detection-0004 | cvat | error | 0 | 1/1
nuclio | openvino.omz.public.faster_rcnn_inception_v2_coco | cvat | error | 0 | 1/1
nuclio | openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco | cvat | error | 0 | 1/1
nuclio | openvino.omz.public.yolo-v3-tf | cvat | error | 0 | 1/1
nuclio | openvino.omz.semantic-segmentation-adas-0001 | cvat | error | 0 | 1/1
nuclio | tf.faster_rcnn_inception_v2_coco | cvat | ready | 39731 | 1/1
nuclio | tf.matterport.mask_rcnn | cvat | ready | 37237 | 1/1

docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ce454f5ed371 cvat/tf.faster_rcnn_inception_v2_coco:latest "processor" 44 minutes ago Up 44 minutes (healthy) 0.0.0.0:39731->8080/tcp nuclio-nuclio-tf.faster_rcnn_inception_v2_coco
dde7ad667e3b cvat/tf.matterport.mask_rcnn:latest "processor" 45 minutes ago Up 45 minutes (healthy) 0.0.0.0:37237->8080/tcp nuclio-nuclio-tf.matterport.mask_rcnn
2a9b80907c35 cvat/openvino.dextr:latest "processor" 46 minutes ago Restarting (1) 44 seconds ago nuclio-nuclio-openvino.dextr
21170f4a7916 cvat/openvino.omz.intel.person-reidentification-retail-0300:latest "processor" 48 minutes ago Restarting (1) 3 seconds ago nuclio-nuclio-openvino.omz.intel.person-reidentification-retail-0300
13b9d8996523 cvat/openvino.omz.intel.semantic-segmentation-adas-0001:latest "processor" 49 minutes ago Restarting (1) 28 seconds ago nuclio-nuclio-openvino.omz.semantic-segmentation-adas-0001
81c6d6abd728 cvat/openvino.omz.intel.text-detection-0004:latest "processor" 50 minutes ago Restarting (1) 57 seconds ago nuclio-nuclio-openvino.omz.intel.text-detection-0004
13f30e51675d cvat/openvino.omz.public.yolo-v3-tf:latest "processor" 52 minutes ago Restarting (1) 20 seconds ago nuclio-nuclio-openvino.omz.public.yolo-v3-tf
4e454024a67b cvat/openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco:latest "processor" 53 minutes ago Restarting (1) 41 seconds ago nuclio-nuclio-openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco
86900a751357 cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest "processor" 55 minutes ago Restarting (1) 6 seconds ago nuclio-nuclio-openvino.omz.public.faster_rcnn_inception_v2_coco
c096e1d34a9f alpine:3.11 "/bin/sh -c '/bin/sl…" 3 hours ago Up 3 hours nuclio-local-storage-reader
10761a37f44d nginx:stable-alpine "/docker-entrypoint.…" 9 hours ago Up 9 hours 0.0.0.0:8080->80/tcp cvat_proxy
888326330e63 cvat/server "/usr/bin/supervisord" 9 hours ago Up 9 hours 8080/tcp, 8443/tcp cvat
a82bd68fdccd cvat_logstash "/usr/local/bin/dock…" 9 hours ago Up 9 hours 5000/tcp, 5044/tcp, 9600/tcp cvat_logstash
a89ce4f4ad30 cvat_kibana "/usr/local/bin/kiba…" 9 hours ago Up 9 hours 5601/tcp cvat_kibana
1f55c7d21943 cvat_elasticsearch "/usr/local/bin/dock…" 9 hours ago Up 9 hours 9200/tcp, 9300/tcp cvat_elasticsearch
4117c788c958 cvat/ui "/docker-entrypoint.…" 9 hours ago Up 9 hours 80/tcp cvat_ui
0c50074a51c5 quay.io/nuclio/dashboard:1.4.8-amd64 "sh -c ./runner.sh" 9 hours ago Up 9 hours 80/tcp, 0.0.0.0:8070->8070/tcp nuclio

nuctl version
nuctl-1.4.14-linux-amd64
@nmanovic nmanovic self-assigned this Aug 3, 2020
@nmanovic nmanovic added the bug Something isn't working label Aug 3, 2020
@nmanovic nmanovic added this to the 1.1.0-release milestone Aug 3, 2020
@nmanovic
Copy link
Contributor

nmanovic commented Aug 3, 2020

@creativesiva , could you please provide the full deployment log for functions? Something like I did below:

20.08.03 16:49:26.992                     nuctl (I) Deploying function {"name": ""}
20.08.03 16:49:26.993                     nuctl (I) Building {"versionInfo": "Label: 1.4.8, Git commit: 238d4539ac7783896d6c414535d0462b5f4cbcf1, OS: darwin, Arch: amd64, Go version: go1.14.3", "name": ""}
20.08.03 16:49:27.199                     nuctl (I) Cleaning up before deployment
20.08.03 16:49:27.284                     nuctl (I) Function already exists, deleting
20.08.03 16:49:29.836                     nuctl (I) Staging files and preparing base images
20.08.03 16:49:29.840                     nuctl (I) Building processor image {"imageName": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest"}
20.08.03 16:49:29.840     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.4.8-amd64"}
20.08.03 16:49:35.541     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"}
20.08.03 16:49:39.253            nuctl.platform (I) Building docker image {"image": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest"}
20.08.03 16:49:40.483            nuctl.platform (I) Pushing docker image into registry {"image": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest", "registry": ""}
20.08.03 16:49:40.483            nuctl.platform (I) Docker image was successfully built and pushed into docker registry {"image": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest"}
20.08.03 16:49:40.483                     nuctl (I) Build complete {"result": {"Image":"cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest","UpdatedFunctionConfig":{"metadata":{"name":"openvino.omz.public.faster_rcnn_inception_v2_coco","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"framework":"openvino","name":"Faster RCNN","spec":"[\n  { \"id\": 1, \"name\": \"person\" },\n  { \"id\": 2, \"name\": \"bicycle\" },\n  { \"id\": 3, \"name\": \"car\" },\n  { \"id\": 4, \"name\": \"motorcycle\" },\n  { \"id\": 5, \"name\": \"airplane\" },\n  { \"id\": 6, \"name\": \"bus\" },\n  { \"id\": 7, \"name\": \"train\" },\n  { \"id\": 8, \"name\": \"truck\" },\n  { \"id\": 9, \"name\": \"boat\" },\n  { \"id\":10, \"name\": \"traffic_light\" },\n  { \"id\":11, \"name\": \"fire_hydrant\" },\n  { \"id\":13, \"name\": \"stop_sign\" },\n  { \"id\":14, \"name\": \"parking_meter\" },\n  { \"id\":15, \"name\": \"bench\" },\n  { \"id\":16, \"name\": \"bird\" },\n  { \"id\":17, \"name\": \"cat\" },\n  { \"id\":18, \"name\": \"dog\" },\n  { \"id\":19, \"name\": \"horse\" },\n  { \"id\":20, \"name\": \"sheep\" },\n  { \"id\":21, \"name\": \"cow\" },\n  { \"id\":22, \"name\": \"elephant\" },\n  { \"id\":23, \"name\": \"bear\" },\n  { \"id\":24, \"name\": \"zebra\" },\n  { \"id\":25, \"name\": \"giraffe\" },\n  { \"id\":27, \"name\": \"backpack\" },\n  { \"id\":28, \"name\": \"umbrella\" },\n  { \"id\":31, \"name\": \"handbag\" },\n  { \"id\":32, \"name\": \"tie\" },\n  { \"id\":33, \"name\": \"suitcase\" },\n  { \"id\":34, \"name\": \"frisbee\" },\n  { \"id\":35, \"name\": \"skis\" },\n  { \"id\":36, \"name\": \"snowboard\" },\n  { \"id\":37, \"name\": \"sports_ball\" },\n  { \"id\":38, \"name\": \"kite\" },\n  { \"id\":39, \"name\": \"baseball_bat\" },\n  { \"id\":40, \"name\": \"baseball_glove\" },\n  { \"id\":41, \"name\": \"skateboard\" },\n  { \"id\":42, \"name\": \"surfboard\" },\n  { \"id\":43, \"name\": \"tennis_racket\" },\n  { \"id\":44, \"name\": \"bottle\" },\n  { \"id\":46, \"name\": \"wine_glass\" },\n  { \"id\":47, \"name\": \"cup\" },\n  { \"id\":48, \"name\": \"fork\" },\n  { \"id\":49, \"name\": \"knife\" },\n  { \"id\":50, \"name\": \"spoon\" },\n  { \"id\":51, \"name\": \"bowl\" },\n  { \"id\":52, \"name\": \"banana\" },\n  { \"id\":53, \"name\": \"apple\" },\n  { \"id\":54, \"name\": \"sandwich\" },\n  { \"id\":55, \"name\": \"orange\" },\n  { \"id\":56, \"name\": \"broccoli\" },\n  { \"id\":57, \"name\": \"carrot\" },\n  { \"id\":58, \"name\": \"hot_dog\" },\n  { \"id\":59, \"name\": \"pizza\" },\n  { \"id\":60, \"name\": \"donut\" },\n  { \"id\":61, \"name\": \"cake\" },\n  { \"id\":62, \"name\": \"chair\" },\n  { \"id\":63, \"name\": \"couch\" },\n  { \"id\":64, \"name\": \"potted_plant\" },\n  { \"id\":65, \"name\": \"bed\" },\n  { \"id\":67, \"name\": \"dining_table\" },\n  { \"id\":70, \"name\": \"toilet\" },\n  { \"id\":72, \"name\": \"tv\" },\n  { \"id\":73, \"name\": \"laptop\" },\n  { \"id\":74, \"name\": \"mouse\" },\n  { \"id\":75, \"name\": \"remote\" },\n  { \"id\":76, \"name\": \"keyboard\" },\n  { \"id\":77, \"name\": \"cell_phone\" },\n  { \"id\":78, \"name\": \"microwave\" },\n  { \"id\":79, \"name\": \"oven\" },\n  { \"id\":80, \"name\": \"toaster\" },\n  { \"id\":81, \"name\": \"sink\" },\n  { \"id\":83, \"name\": \"refrigerator\" },\n  { \"id\":84, \"name\": \"book\" },\n  { \"id\":85, \"name\": \"clock\" },\n  { \"id\":86, \"name\": \"vase\" },\n  { \"id\":87, \"name\": \"scissors\" },\n  { \"id\":88, \"name\": \"teddy_bear\" },\n  { \"id\":89, \"name\": \"hair_drier\" },\n  { \"id\":90, \"name\": \"toothbrush\" }\n]\n","type":"detector"}},"spec":{"description":"Faster RCNN inception v2 COCO via Intel OpenVINO toolkit","handler":"main:handler","runtime":"python:3.6","env":[{"name":"NUCLIO_PYTHON_EXE_PATH","value":"/opt/nuclio/common/python3"}],"resources":{},"image":"cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest","targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"","maxWorkers":2,"workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432}}},"volumes":[{"volume":{"name":"volume-1","hostPath":{"path":"/Users/nmanovic/Workspace/cvat/serverless/openvino/common"}},"volumeMount":{"name":"volume-1","mountPath":"/opt/nuclio/common"}}],"build":{"image":"cvat/openvino.omz.public.faster_rcnn_inception_v2_coco","baseImage":"openvino/ubuntu18_dev:2020.2","directives":{"postCopy":[{"kind":"USER","value":"openvino"}],"preCopy":[{"kind":"USER","value":"root"},{"kind":"WORKDIR","value":"/opt/nuclio"},{"kind":"RUN","value":"ln -s /usr/bin/pip3 /usr/bin/pip"},{"kind":"RUN","value":"/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name faster_rcnn_inception_v2_coco -o /opt/nuclio/open_model_zoo"},{"kind":"RUN","value":"/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo"}]},"codeEntryType":"image"},"platform":{"attributes":{"restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":60,"eventTimeout":"30s"}}}}
20.08.03 16:49:40.908            nuctl.platform (I) Waiting for function to be ready {"timeout": 60}
20.08.03 16:49:43.053                     nuctl (I) Function deploy complete {"httpPort": 54616}
  NAMESPACE |                             NAME                              | PROJECT | STATE | NODE PORT | REPLICAS
  nuclio    | openvino.omz.intel.text-detection-0004                        | cvat    | ready |     54900 | 1/1
  nuclio    | openvino.omz.public.faster_rcnn_inception_v2_coco             | cvat    | ready |     54616 | 1/1
  nuclio    | openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco | cvat    | ready |     54716 | 1/1
  nuclio    | openvino.omz.public.yolo-v3-tf                                | cvat    | error |     57308 | 1/1
  nuclio    | openvino.omz.semantic-segmentation-adas-0001                  | cvat    | ready |     54992 | 1/1
  nuclio    | tf.faster_rcnn_inception_v2_coco                              | cvat    | ready |     55372 | 1/1

@creativesiva
Copy link
Author

@nmanovic , Below is the deployment log.

error_log.txt

20.08.03 22:26:11.277 nuctl.platform (I) Waiting for function to be ready {"timeout": 60}
20.08.03 22:26:12.568 nuctl (I) Function deploy complete {"functionName": "tf.matterport.mask_rcnn", "httpPort": 37237}
20.08.03 22:26:13.531 nuctl (I) Deploying function {"name": ""}
20.08.03 22:26:13.532 nuctl (I) Building {"versionInfo": "Label: 1.4.14, Git commit: e10e9fb123caafbe4f95310a0e2ccfc71368ba87, OS: linux, Arch: amd64, Go version: go1.14.3", "name": ""}
20.08.03 22:26:13.767 nuctl (I) Cleaning up before deployment
20.08.03 22:26:13.880 nuctl (I) Function already exists, deleting
20.08.03 22:26:15.712 nuctl (I) Staging files and preparing base images
20.08.03 22:26:15.714 nuctl (I) Building processor image {"imageName": "cvat/tf.faster_rcnn_inception_v2_coco:latest"}
20.08.03 22:26:15.714 nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.4.14-amd64"}
20.08.03 22:26:22.932 nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"}
20.08.03 22:26:34.942 nuctl.platform (I) Building docker image {"image": "cvat/tf.faster_rcnn_inception_v2_coco:latest"}
20.08.03 22:26:35.301 nuctl.platform (I) Pushing docker image into registry {"image": "cvat/tf.faster_rcnn_inception_v2_coco:latest", "registry": ""}
20.08.03 22:26:35.301 nuctl.platform (I) Docker image was successfully built and pushed into docker registry {"image": "cvat/tf.faster_rcnn_inception_v2_coco:latest"}
20.08.03 22:26:35.301 nuctl (I) Build complete {"result": {"Image":"cvat/tf.faster_rcnn_inception_v2_coco:latest","UpdatedFunctionConfig":{"metadata":{"name":"tf.faster_rcnn_inception_v2_coco","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"framework":"tensorflow","name":"Faster RCNN via Tensorflow","spec":"[\n { "id": 1, "name": "person" },\n { "id": 2, "name": "bicycle" },\n { "id": 3, "name": "car" },\n { "id": 4, "name": "motorcycle" },\n { "id": 5, "name": "airplane" },\n { "id": 6, "name": "bus" },\n { "id": 7, "name": "train" },\n { "id": 8, "name": "truck" },\n { "id": 9, "name": "boat" },\n { "id":10, "name": "traffic_light" },\n { "id":11, "name": "fire_hydrant" },\n { "id":13, "name": "stop_sign" },\n { "id":14, "name": "parking_meter" },\n { "id":15, "name": "bench" },\n { "id":16, "name": "bird" },\n { "id":17, "name": "cat" },\n { "id":18, "name": "dog" },\n { "id":19, "name": "horse" },\n { "id":20, "name": "sheep" },\n { "id":21, "name": "cow" },\n { "id":22, "name": "elephant" },\n { "id":23, "name": "bear" },\n { "id":24, "name": "zebra" },\n { "id":25, "name": "giraffe" },\n { "id":27, "name": "backpack" },\n { "id":28, "name": "umbrella" },\n { "id":31, "name": "handbag" },\n { "id":32, "name": "tie" },\n { "id":33, "name": "suitcase" },\n { "id":34, "name": "frisbee" },\n { "id":35, "name": "skis" },\n { "id":36, "name": "snowboard" },\n { "id":37, "name": "sports_ball" },\n { "id":38, "name": "kite" },\n { "id":39, "name": "baseball_bat" },\n { "id":40, "name": "baseball_glove" },\n { "id":41, "name": "skateboard" },\n { "id":42, "name": "surfboard" },\n { "id":43, "name": "tennis_racket" },\n { "id":44, "name": "bottle" },\n { "id":46, "name": "wine_glass" },\n { "id":47, "name": "cup" },\n { "id":48, "name": "fork" },\n { "id":49, "name": "knife" },\n { "id":50, "name": "spoon" },\n { "id":51, "name": "bowl" },\n { "id":52, "name": "banana" },\n { "id":53, "name": "apple" },\n { "id":54, "name": "sandwich" },\n { "id":55, "name": "orange" },\n { "id":56, "name": "broccoli" },\n { "id":57, "name": "carrot" },\n { "id":58, "name": "hot_dog" },\n { "id":59, "name": "pizza" },\n { "id":60, "name": "donut" },\n { "id":61, "name": "cake" },\n { "id":62, "name": "chair" },\n { "id":63, "name": "couch" },\n { "id":64, "name": "potted_plant" },\n { "id":65, "name": "bed" },\n { "id":67, "name": "dining_table" },\n { "id":70, "name": "toilet" },\n { "id":72, "name": "tv" },\n { "id":73, "name": "laptop" },\n { "id":74, "name": "mouse" },\n { "id":75, "name": "remote" },\n { "id":76, "name": "keyboard" },\n { "id":77, "name": "cell_phone" },\n { "id":78, "name": "microwave" },\n { "id":79, "name": "oven" },\n { "id":80, "name": "toaster" },\n { "id":81, "name": "sink" },\n { "id":83, "name": "refrigerator" },\n { "id":84, "name": "book" },\n { "id":85, "name": "clock" },\n { "id":86, "name": "vase" },\n { "id":87, "name": "scissors" },\n { "id":88, "name": "teddy_bear" },\n { "id":89, "name": "hair_drier" },\n { "id":90, "name": "toothbrush" }\n]\n","type":"detector"}},"spec":{"description":"Faster RCNN from Tensorflow Object Detection API","handler":"main:handler","runtime":"python:3.6","resources":{},"image":"cvat/tf.faster_rcnn_inception_v2_coco:latest","targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"","maxWorkers":2,"workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432}}},"build":{"image":"cvat/tf.faster_rcnn_inception_v2_coco","baseImage":"tensorflow/tensorflow:2.1.1","directives":{"postCopy":[{"kind":"RUN","value":"curl -O http://download.tensorflow.org/models/object_detection/faster_rcnn_inception_v2_coco_2018_01_28.tar.gz"},{"kind":"RUN","value":"tar -xzf faster_rcnn_inception_v2_coco_2018_01_28.tar.gz \u0026\u0026 rm faster_rcnn_inception_v2_coco_2018_01_28.tar.gz"},{"kind":"RUN","value":"ln -s faster_rcnn_inception_v2_coco_2018_01_28 faster_rcnn"},{"kind":"RUN","value":"pip install pillow pyyaml"}],"preCopy":[{"kind":"RUN","value":"apt install curl"},{"kind":"WORKDIR","value":"/opt/nuclio"}]},"codeEntryType":"image"},"platform":{"attributes":{"restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":60,"eventTimeout":"30s"}}}}
20.08.03 22:26:37.812 nuctl.platform (I) Waiting for function to be ready {"timeout": 60}
20.08.03 22:26:39.205 nuctl (I) Function deploy complete {"functionName": "tf.faster_rcnn_inception_v2_coco", "httpPort": 39731}
NAMESPACE | NAME | PROJECT | STATE | NODE PORT | REPLICAS
nuclio | openvino.dextr | cvat | error | 0 | 1/1
nuclio | openvino.omz.intel.person-reidentification-retail-0300 | cvat | error | 0 | 1/1
nuclio | openvino.omz.intel.text-detection-0004 | cvat | error | 0 | 1/1
nuclio | openvino.omz.public.faster_rcnn_inception_v2_coco | cvat | error | 0 | 1/1
nuclio | openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco | cvat | error | 0 | 1/1
nuclio | openvino.omz.public.yolo-v3-tf | cvat | error | 0 | 1/1
nuclio | openvino.omz.semantic-segmentation-adas-0001 | cvat | error | 0 | 1/1
nuclio | tf.faster_rcnn_inception_v2_coco | cvat | ready | 39731 | 1/1
nuclio | tf.matterport.mask_rcnn | cvat | ready | 37237 | 1/1

@nmanovic
Copy link
Contributor

nmanovic commented Aug 4, 2020

@creativesiva , I have updated the documentation. Could you please add --platform local command line argument when you deploy a function (deploy.sh already has the additional argument)? Please notify me if it solves the problem.

For further investigation please add --verbose flag to a nuclio deploy command. It will provide additional information.

Also it looks like need more information about your system. Which OS do you use? Please provide as much details about your system as possible. For example, docker version.

@AJ-RR
Copy link

AJ-RR commented Aug 4, 2020

I have the same issue. I am using Ubuntu 18.04.1 and docker version 19.03.6. Adding --platform local command line argument does not solve the issue.

@nmanovic
Copy link
Contributor

nmanovic commented Aug 4, 2020

@AJ-RR , could you please also publish your deployment log?

@creativesiva
Copy link
Author

creativesiva commented Aug 4, 2020 •
edited
Loading

@creativesiva , I have updated the documentation. Could you please add --platform local command line argument when you deploy a function (deploy.sh already has the additional argument)? Please notify me if it solves the problem.
For further investigation please add --verbose flag to a nuclio deploy command. It will provide additional information.

Also it looks like need more information about your system. Which OS do you use? Please provide as much details about your system as possible. For example, docker version.

nuctl_deploy_verbose_log.txt

I have tried with --platfom local, Hard Luck.
OS:Ubuntu 18.04.4, Docker-version: 19.03.12

@AJ-RR
Copy link

AJ-RR commented Aug 4, 2020 •
edited
Loading

deployment_log.txt
@nmanovic, Here it is
Error - Container wasn't healthy in time
/nuclio/pkg/dockerclient/shell.go:429

Call stack:
Container wasn't healthy in time
/nuclio/pkg/dockerclient/shell.go:429
Function wasn't ready in time. Logs:

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
/nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
/nuclio/cmd/processor/app/processor.go:265

/nuclio/pkg/platform/local/platform.go:628

Failed to deploy function
...//nuclio/pkg/platform/abstract/platform.go:171

@nmanovic nmanovic linked a pull request Aug 5, 2020 that will close this issue
Siammask tracker as DL serverless function #1988
Merged
8 tasks
@nmanovic
Copy link
Contributor

nmanovic commented Aug 5, 2020

@creativesiva , we are investigating the issue. I cannot reproduce it on my Linux and Mac machines. But an internal team also reported the problem. I know that nuclio guys also are investigating the issue. I hope to find a solution soon. If you can help and investigate on your end, it will be perfect.

@nmanovic nmanovic changed the title Issue loading openvino and pytorch sample models in Semi-automatic and automatic annotation Issue loading openvino models for semi-automatic and automatic annotation Aug 5, 2020
@nmanovic nmanovic linked a pull request Aug 6, 2020 that will close this issue
Workaround a problem with deployment openvino serverless functions #1996
Merged
8 tasks
@nmanovic
Copy link
Contributor

nmanovic commented Aug 6, 2020

@creativesiva , could you please try a workaround in PR #1996?

@creativesiva
Copy link
Author

@creativesiva , could you please try a workaround in PR #1996?

@nmanovic , Please suggest how can i check this!!, I am not sure, what to do!

@creativesiva
Copy link
Author

I understand that you have removed "- kind: USER value: openvino" from function.yml and did the same in my local repository. Now it is showing different error.
my Command
sudo ./nuctl deploy --project-name cvat --path serverless/openvino/omz/intel/semantic-segmentation-adas-0001/nuclio --volume serverless/openvino/common:/opt/nuclio/common --platform local
Error Log

20.08.06 23:13:21.888                     nuctl (I) Deploying function {"name": ""}
20.08.06 23:13:21.888                     nuctl (I) Building {"versionInfo": "Label: 1.4.14, Git commit: e10e9fb123caafbe4f95310a0e2ccfc71368ba87, OS: linux, Arch: amd64, Go version: go1.14.3", "name": ""}
20.08.06 23:13:22.146                     nuctl (I) Cleaning up before deployment
20.08.06 23:13:22.209                     nuctl (I) Staging files and preparing base images
20.08.06 23:13:22.210                     nuctl (I) Building processor image {"imageName": "cvat/openvino.omz.intel.semantic-segmentation-adas-0001:latest"}
20.08.06 23:13:22.210     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.4.14-amd64"}
20.08.06 23:13:33.632     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"}
20.08.06 23:13:46.855            nuctl.platform (I) Building docker image {"image": "cvat/openvino.omz.intel.semantic-segmentation-adas-0001:latest"}
20.08.06 23:13:47.649            nuctl.platform (I) Pushing docker image into registry {"image": "cvat/openvino.omz.intel.semantic-segmentation-adas-0001:latest", "registry": ""}
20.08.06 23:13:47.649            nuctl.platform (I) Docker image was successfully built and pushed into docker registry {"image": "cvat/openvino.omz.intel.semantic-segmentation-adas-0001:latest"}
20.08.06 23:13:47.649                     nuctl (I) Build complete {"result": {"Image":"cvat/openvino.omz.intel.semantic-segmentation-adas-0001:latest","UpdatedFunctionConfig":{"metadata":{"name":"openvino.omz.semantic-segmentation-adas-0001","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"framework":"openvino","name":"Semantic segmentation for ADAS","spec":"[\n  { \"id\": 0, \"name\": \"road\" },\n  { \"id\": 1, \"name\": \"sidewalk\" },\n  { \"id\": 2, \"name\": \"building\" },\n  { \"id\": 3, \"name\": \"wall\" },\n  { \"id\": 4, \"name\": \"fence\" },\n  { \"id\": 5, \"name\": \"pole\" },\n  { \"id\": 6, \"name\": \"traffic light\" },\n  { \"id\": 7, \"name\": \"traffic sign\" },\n  { \"id\": 8, \"name\": \"vegetation\" },\n  { \"id\": 9, \"name\": \"terrain\" },\n  { \"id\": 10, \"name\": \"sky\" },\n  { \"id\": 11, \"name\": \"person\" },\n  { \"id\": 12, \"name\": \"rider\" },\n  { \"id\": 13, \"name\": \"car\" },\n  { \"id\": 14, \"name\": \"truck\" },\n  { \"id\": 15, \"name\": \"bus\" },\n  { \"id\": 16, \"name\": \"train\" },\n  { \"id\": 17, \"name\": \"motorcycle\" },\n  { \"id\": 18, \"name\": \"bicycle\" },\n  { \"id\": 19, \"name\": \"ego-vehicle\" },\n  { \"id\": 20, \"name\": \"background\" }\n]\n","type":"detector"}},"spec":{"description":"Segmentation network to classify each pixel into typical 20 classes for ADAS","handler":"main:handler","runtime":"python:3.6","env":[{"name":"NUCLIO_PYTHON_EXE_PATH","value":"/opt/nuclio/common/python3"}],"resources":{},"image":"cvat/openvino.omz.intel.semantic-segmentation-adas-0001:latest","targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"","maxWorkers":2,"workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432}}},"volumes":[{"volume":{"name":"volume-1","hostPath":{"path":"serverless/openvino/common"}},"volumeMount":{"name":"volume-1","mountPath":"/opt/nuclio/common"}}],"build":{"image":"cvat/openvino.omz.intel.semantic-segmentation-adas-0001","baseImage":"openvino/ubuntu18_dev:2020.2","directives":{"postCopy":[{"kind":"RUN","value":"apt update \u0026\u0026 DEBIAN_FRONTEND=noninteractive apt install --no-install-recommends -y python3-skimage"},{"kind":"RUN","value":"pip3 install \"numpy\u003c1.16.0\""}],"preCopy":[{"kind":"USER","value":"root"},{"kind":"WORKDIR","value":"/opt/nuclio"},{"kind":"RUN","value":"ln -s /usr/bin/pip3 /usr/bin/pip"},{"kind":"RUN","value":"/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name semantic-segmentation-adas-0001 -o /opt/nuclio/open_model_zoo"}]},"codeEntryType":"image"},"platform":{"attributes":{"restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":60,"eventTimeout":"30s"}}}}
20.08.06 23:13:47.979     nuctl.platform.docker (W) Failed to run container {"err": "stdout:\ndocker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n\nstderr:\n", "errVerbose": "\nError - exit status 125\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\ndocker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n\nstderr:\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nstdout:\ndocker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n\nstderr:\n", "errCauses": [{"error": "exit status 125"}], "stdout": "docker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n", "stderr": ""}
20.08.06 23:13:47.979                     nuctl (W) Create function failed, setting function status {"err": "Failed to run docker container", "errVerbose": "\nError - exit status 125\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\ndocker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n\nstderr:\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nFailed to run docker container\n    /nuclio/pkg/platform/local/platform.go:605\nFailed to run docker container", "errCauses": [{"error": "stdout:\ndocker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n\nstderr:\n", "errorVerbose": "\nError - exit status 125\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\ndocker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n\nstderr:\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nstdout:\ndocker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n\nstderr:\n", "errorCauses": [{"error": "exit status 125"}]}]}

Error - exit status 125
    /nuclio/pkg/cmdrunner/cmdrunner.go:124

Call stack:
stdout:
docker: Error response from daemon: create serverless/openvino/common: "serverless/openvino/common" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
See 'docker run --help'.

stderr:

    /nuclio/pkg/cmdrunner/cmdrunner.go:124
Failed to run docker container
    /nuclio/pkg/platform/local/platform.go:605
Failed to deploy function
    ...//nuclio/pkg/platform/abstract/platform.go:171

@nmanovic
Copy link
Contributor

nmanovic commented Aug 7, 2020 •
edited
Loading

@creativesiva , the mentioned deploy command isn't correct. The error message is clear here: If you intended to pass a host directory, use absolute path. For volumes you have to use absolute paths: ./nuctl deploy --project-name cvat --path serverless/openvino/omz/intel/semantic-segmentation-adas-0001/nuclio --volume <abspath>/serverless/openvino/common:/opt/nuclio/common --platform local

Also you don't need to use sudo if your docker setup is correct (you are in docker group).

@nmanovic nmanovic reopened this Aug 7, 2020
@nmanovic nmanovic mentioned this issue Aug 7, 2020
Closed
@creativesiva
Copy link
Author

@creativesiva , the mentioned deploy command isn't correct. The error message is clear here: If you intended to pass a host directory, use absolute path. For volumes you have to use absolute paths: ./nuctl deploy --project-name cvat --path serverless/openvino/omz/intel/semantic-segmentation-adas-0001/nuclio --volume <abspath>/serverless/openvino/common:/opt/nuclio/common --platform local

Also you don't need to use sudo if your docker setup is correct (you are in docker group).

@nmanovic, Thanks. The model deployment is successful now. (I have not made changes to text-detection function, that is why error, rest all functions are modified as per your suggestion)

Thanks a lot.

  NAMESPACE |                             NAME                              | PROJECT | STATE | NODE PORT | REPLICAS  
  nuclio    | openvino.dextr                                                | cvat    | ready |     46667 | 1/1       
  nuclio    | openvino.omz.intel.person-reidentification-retail-0300        | cvat    | ready |     35367 | 1/1       
  nuclio    | openvino.omz.intel.text-detection-0004                        | cvat    | error |         0 | 1/1       
  nuclio    | openvino.omz.public.faster_rcnn_inception_v2_coco             | cvat    | ready |     34845 | 1/1       
  nuclio    | openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco | cvat    | ready |     36551 | 1/1       
  nuclio    | openvino.omz.public.yolo-v3-tf                                | cvat    | ready |     37233 | 1/1       
  nuclio    | openvino.omz.semantic-segmentation-adas-0001                  | cvat    | ready |     33293 | 1/1       
  nuclio    | tf.faster_rcnn_inception_v2_coco                              | cvat    | ready |     39731 | 1/1       
  nuclio    | tf.matterport.mask_rcnn                                       | cvat    | ready |     37237 | 1/1

@nmanovic
Copy link
Contributor

nmanovic commented Aug 7, 2020

@creativesiva , could you please help me to test a fix in nuclio? Below I have attached nuctl for linux with a fix which just adds necessary permissions to /tmp/processor-config-* file (liranbg/nuclio#22). Please revert all changes inside CVAT and try the binary to deploy functions. Does it work? I cannot check on my end unfortunately.

nuctl-latest-linux-amd64.zip

@nmanovic nmanovic mentioned this issue Aug 7, 2020
Merged
8 tasks
@creativesiva
Copy link
Author

@nmanovic , i have reverted function.yml changes and used nuctl-latest-linux-amd64 to run the deploy.sh. It is throwing error.
Error Log

20.08.07 21:18:49.114                     nuctl (I) Deploying function {"name": ""}
20.08.07 21:18:49.114                     nuctl (I) Building {"versionInfo": "Label: 1.4.14, Git commit: e10e9fb123caafbe4f95310a0e2ccfc71368ba87, OS: linux, Arch: amd64, Go version: go1.14.3", "name": ""}
20.08.07 21:18:49.364                     nuctl (I) Cleaning up before deployment
20.08.07 21:18:49.475                     nuctl (I) Function already exists, deleting
20.08.07 21:18:50.691                     nuctl (I) Staging files and preparing base images
20.08.07 21:18:50.692                     nuctl (I) Building processor image {"imageName": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest"}
20.08.07 21:18:50.692     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.4.14-amd64"}
20.08.07 21:18:57.453     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"}
20.08.07 21:19:10.244            nuctl.platform (I) Building docker image {"image": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest"}
20.08.07 21:19:27.725            nuctl.platform (I) Pushing docker image into registry {"image": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest", "registry": ""}
20.08.07 21:19:27.725            nuctl.platform (I) Docker image was successfully built and pushed into docker registry {"image": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest"}
20.08.07 21:19:27.725                     nuctl (I) Build complete {"result": {"Image":"cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest","UpdatedFunctionConfig":{"metadata":{"name":"openvino.omz.public.faster_rcnn_inception_v2_coco","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"framework":"openvino","name":"Faster RCNN","spec":"[\n  { \"id\": 1, \"name\": \"person\" },\n  { \"id\": 2, \"name\": \"bicycle\" },\n  { \"id\": 3, \"name\": \"car\" },\n  { \"id\": 4, \"name\": \"motorcycle\" },\n  { \"id\": 5, \"name\": \"airplane\" },\n  { \"id\": 6, \"name\": \"bus\" },\n  { \"id\": 7, \"name\": \"train\" },\n  { \"id\": 8, \"name\": \"truck\" },\n  { \"id\": 9, \"name\": \"boat\" },\n  { \"id\":10, \"name\": \"traffic_light\" },\n  { \"id\":11, \"name\": \"fire_hydrant\" },\n  { \"id\":13, \"name\": \"stop_sign\" },\n  { \"id\":14, \"name\": \"parking_meter\" },\n  { \"id\":15, \"name\": \"bench\" },\n  { \"id\":16, \"name\": \"bird\" },\n  { \"id\":17, \"name\": \"cat\" },\n  { \"id\":18, \"name\": \"dog\" },\n  { \"id\":19, \"name\": \"horse\" },\n  { \"id\":20, \"name\": \"sheep\" },\n  { \"id\":21, \"name\": \"cow\" },\n  { \"id\":22, \"name\": \"elephant\" },\n  { \"id\":23, \"name\": \"bear\" },\n  { \"id\":24, \"name\": \"zebra\" },\n  { \"id\":25, \"name\": \"giraffe\" },\n  { \"id\":27, \"name\": \"backpack\" },\n  { \"id\":28, \"name\": \"umbrella\" },\n  { \"id\":31, \"name\": \"handbag\" },\n  { \"id\":32, \"name\": \"tie\" },\n  { \"id\":33, \"name\": \"suitcase\" },\n  { \"id\":34, \"name\": \"frisbee\" },\n  { \"id\":35, \"name\": \"skis\" },\n  { \"id\":36, \"name\": \"snowboard\" },\n  { \"id\":37, \"name\": \"sports_ball\" },\n  { \"id\":38, \"name\": \"kite\" },\n  { \"id\":39, \"name\": \"baseball_bat\" },\n  { \"id\":40, \"name\": \"baseball_glove\" },\n  { \"id\":41, \"name\": \"skateboard\" },\n  { \"id\":42, \"name\": \"surfboard\" },\n  { \"id\":43, \"name\": \"tennis_racket\" },\n  { \"id\":44, \"name\": \"bottle\" },\n  { \"id\":46, \"name\": \"wine_glass\" },\n  { \"id\":47, \"name\": \"cup\" },\n  { \"id\":48, \"name\": \"fork\" },\n  { \"id\":49, \"name\": \"knife\" },\n  { \"id\":50, \"name\": \"spoon\" },\n  { \"id\":51, \"name\": \"bowl\" },\n  { \"id\":52, \"name\": \"banana\" },\n  { \"id\":53, \"name\": \"apple\" },\n  { \"id\":54, \"name\": \"sandwich\" },\n  { \"id\":55, \"name\": \"orange\" },\n  { \"id\":56, \"name\": \"broccoli\" },\n  { \"id\":57, \"name\": \"carrot\" },\n  { \"id\":58, \"name\": \"hot_dog\" },\n  { \"id\":59, \"name\": \"pizza\" },\n  { \"id\":60, \"name\": \"donut\" },\n  { \"id\":61, \"name\": \"cake\" },\n  { \"id\":62, \"name\": \"chair\" },\n  { \"id\":63, \"name\": \"couch\" },\n  { \"id\":64, \"name\": \"potted_plant\" },\n  { \"id\":65, \"name\": \"bed\" },\n  { \"id\":67, \"name\": \"dining_table\" },\n  { \"id\":70, \"name\": \"toilet\" },\n  { \"id\":72, \"name\": \"tv\" },\n  { \"id\":73, \"name\": \"laptop\" },\n  { \"id\":74, \"name\": \"mouse\" },\n  { \"id\":75, \"name\": \"remote\" },\n  { \"id\":76, \"name\": \"keyboard\" },\n  { \"id\":77, \"name\": \"cell_phone\" },\n  { \"id\":78, \"name\": \"microwave\" },\n  { \"id\":79, \"name\": \"oven\" },\n  { \"id\":80, \"name\": \"toaster\" },\n  { \"id\":81, \"name\": \"sink\" },\n  { \"id\":83, \"name\": \"refrigerator\" },\n  { \"id\":84, \"name\": \"book\" },\n  { \"id\":85, \"name\": \"clock\" },\n  { \"id\":86, \"name\": \"vase\" },\n  { \"id\":87, \"name\": \"scissors\" },\n  { \"id\":88, \"name\": \"teddy_bear\" },\n  { \"id\":89, \"name\": \"hair_drier\" },\n  { \"id\":90, \"name\": \"toothbrush\" }\n]\n","type":"detector"}},"spec":{"description":"Faster RCNN inception v2 COCO via Intel OpenVINO toolkit","handler":"main:handler","runtime":"python:3.6","env":[{"name":"NUCLIO_PYTHON_EXE_PATH","value":"/opt/nuclio/common/python3"}],"resources":{},"image":"cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest","targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"","maxWorkers":2,"workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432}}},"volumes":[{"volume":{"name":"volume-1","hostPath":{"path":"/home/aipl/cvat/serverless/openvino/common"}},"volumeMount":{"name":"volume-1","mountPath":"/opt/nuclio/common"}}],"build":{"image":"cvat/openvino.omz.public.faster_rcnn_inception_v2_coco","baseImage":"openvino/ubuntu18_dev:2020.2","directives":{"postCopy":[{"kind":"USER","value":"openvino"}],"preCopy":[{"kind":"USER","value":"root"},{"kind":"WORKDIR","value":"/opt/nuclio"},{"kind":"RUN","value":"ln -s /usr/bin/pip3 /usr/bin/pip"},{"kind":"RUN","value":"/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name faster_rcnn_inception_v2_coco -o /opt/nuclio/open_model_zoo"},{"kind":"RUN","value":"/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo"}]},"codeEntryType":"image"},"platform":{"attributes":{"restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":60,"eventTimeout":"30s"}}}}
20.08.07 21:19:30.374            nuctl.platform (I) Waiting for function to be ready {"timeout": 60}
20.08.07 21:20:30.423     nuctl.platform.docker (W) Container wasn't healthy within timeout {"containerID": "8280740ef19cc90d2aedb2b962cf1d14c73b36fe42e0fb59578b5cef1cb3d5be", "timeout": "1m0s", "logs": "\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n"}
20.08.07 21:20:30.473                     nuctl (W) Create function failed, setting function status {"err": "Function wasn't ready in time. Logs:\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n", "errVerbose": "\nError - Container wasn't healthy in time\n    /nuclio/pkg/dockerclient/shell.go:429\n\nCall stack:\nContainer wasn't healthy in time\n    /nuclio/pkg/dockerclient/shell.go:429\nFunction wasn't ready in time. Logs:\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\n    /nuclio/pkg/platform/local/platform.go:628\nFunction wasn't ready in time. Logs:\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n", "errCauses": [{"error": "Container wasn't healthy in time", "errorVerbose": "\nError - Container wasn't healthy in time\n    /nuclio/pkg/dockerclient/shell.go:429\n\nCall stack:\nContainer wasn't healthy in time\n    /nuclio/pkg/dockerclient/shell.go:429\nContainer wasn't healthy in time", "errorCauses": [{}]}]}

Error - Container wasn't healthy in time
    /nuclio/pkg/dockerclient/shell.go:429

Call stack:
Container wasn't healthy in time
    /nuclio/pkg/dockerclient/shell.go:429
Function wasn't ready in time. Logs:

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

    /nuclio/pkg/platform/local/platform.go:628
Failed to deploy function
    ...//nuclio/pkg/platform/abstract/platform.go:171

@nmanovic
Copy link
Contributor

nmanovic commented Aug 7, 2020

@creativesiva , thanks for the info. I have merged the PR. Now it should work. I hope nuclio team will be able to reproduce the issue and fix in future releases.

@nmanovic
Copy link
Contributor

nmanovic commented Aug 7, 2020

@creativesiva , are you sure that you used the attached binary? At least Git commit e10e9fb123caafbe4f95310a0e2ccfc71368ba87 in your logs looks strange.

@creativesiva
Copy link
Author

@nmanovic Sorry i did not noticed your question.
Yes, i have used (nuctl-latest-linux-amd64.zip)

@thangvip4321
Copy link

thangvip4321 commented Aug 12, 2020 •
edited
Loading

@nmanovic i have updated my code, but the error still persist.
Here's the log:

I'm running on Ubuntu 18.04, docker version : 19.03.11 , nuctl: 1.4.16.
log_cvat.txt

@nmanovic
Copy link
Contributor

@thangvip4321 , indeed it is another issue: Error - read /etc/nuclio/config/processor/processor.yaml: is a directory

@nmanovic
Copy link
Contributor

@thangvip4321 , let me check with nuctl 1.4.16 if I can reproduce the issue.

@nmanovic
Copy link
Contributor

@thangvip4321 , I cannot reproduce the problem. Need your help.

  • Please create a new issue.
  • Do any functions work on your machine?
  • Run nuctl with --verbose flag and attach logs
  • specify exactly command which you are using

@thangvip4321 thangvip4321 mentioned this issue Aug 12, 2020
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nmanovic nmanovic

Labels
bug Something isn't working
Projects
None yet
Milestone
1.1.0-release
4 participants
@creativesiva @thangvip4321 @nmanovic @AJ-RR