Error while getting preview/thumbnail images

[MauroRibeiroHV]

My actions before raising this issue

• [X ] Read/searched the docs
• [ X] Searched past issues

Hi,

After creating some tasks with a thousand images(many over 10Mb), the task list takes 1 minute to load, it's the timeout for the
https:///api/v1/tasks/4/data?type=preview
request timeout.

Expected Behaviour

When opening the task list, it should open quickly and show a preview in all tasks.

Current Behaviour

It takes 1 minute (timeout on https:///api/v1/tasks/4/data?type=preview) to load the task list and then fails to show the preview image.

Possible Solution

I don't know what is causing this.

Steps to Reproduce (for bugs)

1. Create a task with lots of images (lots with over 10Mb)
2. Create tasks with images bigger than 10Mb
3. Go to Task list
4. It will take 1 minute to open and fail to show previews

Context

This makes listing the tasks to take a long time, and we aren't sure if it only affects that.

Your Environment

• Git hash commit (git log -1):
• CVAT v1.6.0
• Docker version docker version (e.g. Docker 17.0.05):
• Are you using Docker Swarm or Kubernetes?
• Kubernetes
• Operating System and version (e.g. Linux, Windows, MacOS):
• Linux
• Code example or link to GitHub repo or gist to reproduce problem:
• Other diagnostic information / logs:
  Request Method: GET
  Status Code: 504

DEBUG - 2021-10-06 09:22:06,077 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
2021-10-06T09:22:06.083781565Z
2021-10-06 09:28:11,608 DEBG 'rqworker_low' stderr output:
DEBUG - 2021-10-06 09:28:11,605 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-10-06 09:28:11,610 DEBG 'rqworker_low' stderr output:
INFO - 2021-10-06 09:28:11,606 - worker - Cleaning registries for queue: low
2021-10-06T09:28:11.610350578Z
2021-10-06 09:28:35,051 DEBG 'rqworker_default_1' stderr output:
DEBUG - 2021-10-06 09:28:35,049 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-10-06 09:28:35,052 DEBG 'rqworker_default_1' stderr output:
INFO - 2021-10-06 09:28:35,049 - worker - Cleaning registries for queue: default
2021-10-06T09:28:35.052129582Z
2021-10-06 09:28:47,387 DEBG 'runserver' stderr output:
[Wed Oct 06 09:28:47.384893 2021] [wsgi:error] [pid 1227:tid 140289081366272] [client 10.244.2.59:53832] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: https:///tasks

2021-10-06 09:28:47,393 DEBG 'runserver' stderr output:
[Wed Oct 06 09:28:47.393103 2021] [wsgi:error] [pid 1227:tid 140289081091840] [client 10.244.2.59:53834] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: https:///tasks

2021-10-06 09:28:51,178 DEBG 'rqworker_default_0' stderr output:
DEBUG - 2021-10-06 09:28:51,175 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
2021-10-06T09:28:51.178283501Z

Next steps

You may join our Gitter channel for community support.

[azhavoro]

Hi, if I understand correctly, you created tasks with hi-res images, and this case is only specific for such tasks? What is the typical resolution of the images used?

[MauroRibeiroHV]

5472x3078 for example.
I think it's related to jobs with many images, it fails creating the preview image.

[azhavoro]

Thanks, we will try to reproduce on our side.

[syncharny]

@azhavoro Hello, we appear to be experiencing the same issue. Often the thumbnails do not load at all and show as broken. We also see these errors in the server console:

[wsgi:error] [pid 14191:tid 140652084786944] [remote 127.0.0.6:47759] mod_wsgi (pid=14191): Queue timeout expired for WSGI daemon process 'localhost:8080'.

[wsgi:error] [pid 1904:tid 140655035328256] [client 127.0.0.6:40311] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api

Any ideas what may be happening? Most our images are 720p

[Thomasoj]

Hi,
We have the same issue with image previews on an AWS Fargate instance with an EFS volume attached. We only have a few images (about 10, resolution 3024 × 4032) and only 2 tasks. The first requests are fine, but the next requests end with a timeout (we have the timeout set to 90 seconds).

If I disconnect the EFS volume and save the data directly to the Fargate instance, everything is fine and the image previews load correctly.

BTW: same issue is here - #3019

[Rusteam]

Can this lead to CVAT being slow? Does it wait until the timeout before rendering tasks page?

[vgoineau]

I do have same issue here.

We use dev branch in docker-compose file
Traefik is in front of our CVAT deployment

We do have very slow loading. CVAT_UI takes +30 sec to load and we have http 500 or even 504.

If anyone can help us on this issue it would be very appreciated.

Here my cvat_server log :
[Tue Dec 20 00:28:43.931244 2022] [wsgi:error] [pid 69:tid 139700894029568] [client 10.0.2.3:42716] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: https://label..com/tasks?page=1
2022-12-20 00:28:43,931 DEBG 'runserver' stderr output:
[Tue Dec 20 00:28:43.923326 2022] [wsgi:error] [pid 69:tid 139700893763328] [client 10.0.2.3:42686] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: https://label..com/tasks?page=1
[Tue Dec 20 00:28:43.923271 2022] [wsgi:error] [pid 69:tid 139700899600128] [client 10.0.2.3:42710] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: https://label..com/tasks?page=1
2022-12-20 00:28:43,924 DEBG 'runserver' stderr output:
[Tue Dec 20 00:27:43.453858 2022] [wsgi:error] [pid 68:tid 139700373423872] [remote 10.0.2.3:42600] WARNING:django.request:Not Found: /api/user-agreements
2022-12-20 00:27:43,454 DEBG 'runserver' stderr output:
[Tue Dec 20 00:27:43.452589 2022] [wsgi:error] [pid 68:tid 139700373423872] [remote 10.0.2.3:42600] [2022-12-20 00:27:43,452] WARNING django.request: Not Found: /api/user-agreements
[Tue Dec 20 00:26:46.090092 2022] [wsgi:info] [pid 68:tid 139700906622016] mod_wsgi (pid=68, process='localhost:8080', application=''): Loading Python script file '/tmp/cvat-server/handler.wsgi'.
2022-12-20 00:27:43,453 DEBG 'runserver' stderr output:
[Tue Dec 20 00:26:46.089971 2022] [wsgi:info] [pid 68:tid 139700906622016] mod_wsgi (pid=68): Imported 'mod_wsgi'.
[Tue Dec 20 00:26:46.081697 2022] [wsgi:info] [pid 68:tid 139700906622016] mod_wsgi (pid=68): Attach interpreter ''.
2022-12-20 00:26:46,090 DEBG 'runserver' stderr output:

[ryanhulet]

This start after I followed the upgrade instructions from 2.2 to 2.3

[pkumar219]

I do see after upgrade to the latest version and what was the fix that was applied to resolve it ?