502 Bad Gateway

[AndrewOkie]

My actions before raising this issue

• Read/searched the docs
• Searched past issues
• Tried some stupid suggestions like "docker-compose down" or "always run CVAT in a fresh incognito tab"
• Sold my child to Satan

Expected Behaviour

CVAT that works

Current Behaviour

502-s all around. I guess it is something with Traefik not routing to "cvat" container but I'm not sure.
I had the same issue a year ago (it was nginx IIRC). Back then I was able to solve the problem by buying the tool from a competitor and forgetting the pain I went through trying to install this tool on my physical machine.

Steps to Reproduce (yes, for bugs)

1. git clone https://github.com/opencv/cvat
2. cd cvat
3. docker-compose up -d
4. firefox --new-window localhost:8080

Environment

Cloud VM instance with Ubuntu 20.04, Docker CE 20.10. No running containers or other crap that can possibly interfere with CVAT.

Screenshots

Docker Logs

$ docker-compose up
Creating network "cvat_cvat" with the default driver
Creating traefik ... done
Creating cvat_redis ... done
Creating cvat_db ... done
Creating cvat ... done
Creating cvat_ui ... done
Attaching to cvat_db, cvat_redis, traefik, cvat, cvat_ui
cvat_db |
cvat_db | PostgreSQL Database directory appears to contain a database; Skipping initialization
cvat_db |
cvat_db | 2021-10-20 21:36:22.834 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
cvat_db | 2021-10-20 21:36:22.834 UTC [1] LOG: listening on IPv6 address "::", port 5432
cvat_db | 2021-10-20 21:36:22.836 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
cvat_db | 2021-10-20 21:36:22.858 UTC [21] LOG: database system was interrupted; last known up at 2021-10-20 21:28:42 UTC
cvat_db | 2021-10-20 21:36:23.050 UTC [21] LOG: database system was not properly shut down; automatic recovery in progress
cvat_db | 2021-10-20 21:36:23.052 UTC [21] LOG: redo starts at 0/1642488
cvat_db | 2021-10-20 21:36:23.052 UTC [21] LOG: invalid record length at 0/1642568: wanted 24, got 0
cvat_db | 2021-10-20 21:36:23.052 UTC [21] LOG: redo done at 0/1642530
cvat_db | 2021-10-20 21:36:23.070 UTC [1] LOG: database system is ready to accept connections
cvat_redis | 1:C 20 Oct 21:36:22.823 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
cvat_redis | 1:C 20 Oct 21:36:22.823 # Redis version=4.0.14, bits=64, commit=00000000, modified=0, pid=1, just started
cvat_redis | 1:C 20 Oct 21:36:22.823 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
cvat_redis | 1:M 20 Oct 21:36:22.828 * Running mode=standalone, port=6379.
cvat_redis | 1:M 20 Oct 21:36:22.829 # Server initialized
cvat_redis | 1:M 20 Oct 21:36:22.829 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
cvat_redis | 1:M 20 Oct 21:36:22.829 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
cvat_redis | 1:M 20 Oct 21:36:22.829 * Ready to accept connections
cvat_ui | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
cvat_ui | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
cvat_ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
traefik | time="2021-10-20T21:36:23Z" level=info msg="Configuration loaded from flags."
cvat_ui | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
cvat_ui | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf differs from the packaged version
cvat | 2021-10-20 21:36:24,621 INFO RPC interface 'supervisor' initialized
cvat_ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
cvat | 2021-10-20 21:36:24,629 CRIT Server 'unix_http_server' running without any HTTP authentication checking
cvat | 2021-10-20 21:36:24,630 INFO supervisord started with pid 1
cvat_ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
cvat_ui | /docker-entrypoint.sh: Configuration complete; ready for start up
cvat_ui | 2021/10/20 21:36:24 [warn] 1#1: duplicate extension "wasm", content type: "application/wasm", previous content type: "application/wasm" in /etc/nginx/mime.types:98
cvat_ui | nginx: [warn] duplicate extension "wasm", content type: "application/wasm", previous content type: "application/wasm" in /etc/nginx/mime.types:98
cvat_ui | 2021/10/20 21:36:24 [notice] 1#1: using the "epoll" event method
cvat_ui | 2021/10/20 21:36:24 [notice] 1#1: nginx/1.21.3
cvat_ui | 2021/10/20 21:36:24 [notice] 1#1: built by gcc 10.3.1 20210424 (Alpine 10.3.1_git20210424)
cvat_ui | 2021/10/20 21:36:24 [notice] 1#1: OS: Linux 5.4.0-77-generic
cvat_ui | 2021/10/20 21:36:24 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
cvat_ui | 2021/10/20 21:36:24 [notice] 1#1: start worker processes
cvat_ui | 2021/10/20 21:36:24 [notice] 1#1: start worker process 31
cvat | 2021-10-20 21:36:25,632 INFO spawned: 'ssh-agent' with pid 8
cvat | 2021-10-20 21:36:25,635 INFO spawned: 'clamav_update' with pid 9
cvat | 2021-10-20 21:36:25,655 INFO spawned: 'git_status_updater' with pid 10
cvat | 2021-10-20 21:36:25,658 INFO spawned: 'rqscheduler' with pid 11
cvat | 2021-10-20 21:36:25,665 INFO spawned: 'rqworker_default_0' with pid 12
cvat | 2021-10-20 21:36:25,682 INFO spawned: 'rqworker_default_1' with pid 13
cvat | 2021-10-20 21:36:25,713 INFO spawned: 'rqworker_low' with pid 16
cvat | 2021-10-20 21:36:25,738 INFO spawned: 'runserver' with pid 20
cvat | 2021-10-20 21:36:25,763 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139783414488608 for <Subprocess at 139783414487936 with name clamav_update in state STARTING> (stdout)>
cvat | 2021-10-20 21:36:25,771 DEBG fd 14 closed, stopped monitoring <POutputDispatcher at 139783414258176 for <Subprocess at 139783414487936 with name clamav_update in state STARTING> (stderr)>
cvat | 2021-10-20 21:36:25,773 INFO exited: clamav_update (exit status 0; not expected)
cvat | 2021-10-20 21:36:25,776 DEBG received SIGCHLD indicating a child quit
cvat | 2021-10-20 21:36:25,781 DEBG 'ssh-agent' stdout output:
cvat | SSH_AUTH_SOCK=/tmp/ssh-agent.sock; export SSH_AUTH_SOCK;
cvat | echo Agent pid 8;
cvat |
cvat | 2021-10-20 21:36:25,784 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 3 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:36:25,789 DEBG 'git_status_updater' stderr output:
cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout
cvat |
cvat | 2021-10-20 21:36:25,792 DEBG 'rqscheduler' stderr output:
cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout
cvat |
cvat | 2021-10-20 21:36:25,795 DEBG 'rqworker_default_0' stderr output:
cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout
cvat |
cvat | 2021-10-20 21:36:25,797 DEBG 'rqworker_default_1' stderr output:
cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout
cvat |
cvat | 2021-10-20 21:36:25,813 DEBG 'git_status_updater' stderr output:
cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds
cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device
cvat | bash: no job control in this shell
cvat |
cvat | 2021-10-20 21:36:25,822 DEBG 'rqscheduler' stderr output:
cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds
cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device
cvat | bash: no job control in this shell
cvat |
cvat | 2021-10-20 21:36:25,827 DEBG 'rqworker_low' stderr output:
cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout
cvat |
cvat | 2021-10-20 21:36:25,830 DEBG 'rqworker_default_0' stderr output:
cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds
cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device
cvat | bash: no job control in this shell
cvat |
cvat | 2021-10-20 21:36:25,854 DEBG 'runserver' stderr output:
cvat | wait-for-it.sh: waiting for cvat_db:5432 without a timeout
cvat |
cvat | 2021-10-20 21:36:25,861 DEBG 'rqworker_default_1' stderr output:
cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds
cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device
cvat | bash: no job control in this shell
cvat |
cvat | 2021-10-20 21:36:25,903 DEBG 'rqworker_low' stderr output:
cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds
cvat |
cvat | 2021-10-20 21:36:25,905 DEBG 'rqworker_low' stderr output:
cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device
cvat | bash: no job control in this shell
cvat |
cvat_db | 2021-10-20 21:36:25.907 UTC [28] LOG: incomplete startup packet
cvat | 2021-10-20 21:36:25,922 DEBG 'runserver' stderr output:
cvat | wait-for-it.sh: cvat_db:5432 is available after 0 seconds
cvat |
cvat | 2021-10-20 21:36:25,925 DEBG 'runserver' stderr output:
cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device
cvat |
cvat | 2021-10-20 21:36:25,926 DEBG 'runserver' stderr output:
cvat | bash: no job control in this shell
cvat |
cvat | 2021-10-20 21:36:26,473 DEBG 'rqscheduler' stderr output:
cvat | 21:36:26 Registering birth
cvat |
cvat | 2021-10-20 21:36:26,522 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:36:26,526 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:36:26,527 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 11
cvat |
cvat | 2021-10-20 21:36:26,530 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:36:26,534 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 17
cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none)
cvat |
cvat | 2021-10-20 21:36:26,534 DEBG 'rqworker_low' stderr output:
cvat | Identity added: /home/django/.ssh/id_rsa (django@3977e1f9cf92)
cvat |
cvat | 2021-10-20 21:36:26,545 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:36:26,552 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 17
cvat |
cvat | 2021-10-20 21:36:26,555 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none)
cvat |
cvat | 2021-10-20 21:36:26,564 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:36:26,565 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 11
cvat |
cvat | 2021-10-20 21:36:26,582 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:36:26,591 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 17
cvat |
cvat | 2021-10-20 21:36:26,592 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none)
cvat |
cvat | 2021-10-20 21:36:26,602 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:36:26,605 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 11
cvat |
cvat | 2021-10-20 21:36:26,637 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:36:26,638 INFO success: ssh-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
cvat | 2021-10-20 21:36:26,649 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 17
cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none)
cvat |
cvat | 2021-10-20 21:36:26,649 INFO success: git_status_updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
cvat | 2021-10-20 21:36:26,664 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:36:26,665 INFO success: rqscheduler entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
cvat | 2021-10-20 21:36:26,665 INFO success: rqworker_default_0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
cvat | 2021-10-20 21:36:26,670 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 11
cvat |
cvat | 2021-10-20 21:36:26,697 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:36:26,698 INFO success: rqworker_default_1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
cvat | 2021-10-20 21:36:26,708 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 17
cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none)
cvat |
cvat | 2021-10-20 21:36:26,728 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:36:26,729 INFO success: rqworker_low entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
cvat | 2021-10-20 21:36:26,730 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 11
cvat |
cvat | 2021-10-20 21:36:27,733 INFO spawned: 'clamav_update' with pid 89
cvat | 2021-10-20 21:36:27,734 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
cvat | 2021-10-20 21:36:27,768 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 139783414258128 for <Subprocess at 139783414487936 with name clamav_update in state STARTING> (stderr)>
cvat | 2021-10-20 21:36:27,770 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139783414258320 for <Subprocess at 139783414487936 with name clamav_update in state STARTING> (stdout)>
cvat | 2021-10-20 21:36:28,771 INFO success: clamav_update entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
cvat | 2021-10-20 21:36:28,772 INFO exited: clamav_update (exit status 0; expected)
cvat | 2021-10-20 21:36:28,772 DEBG received SIGCHLD indicating a child quit
cvat | 2021-10-20 21:36:52,658 DEBG fd 34 closed, stopped monitoring <POutputDispatcher at 139783413801264 for <Subprocess at 139783414089136 with name rqworker_default_1 in state RUNNING> (stderr)>
cvat | 2021-10-20 21:36:52,668 DEBG fd 30 closed, stopped monitoring <POutputDispatcher at 139783414089184 for <Subprocess at 139783414089136 with name rqworker_default_1 in state RUNNING> (stdout)>
cvat | 2021-10-20 21:36:53,671 INFO exited: rqworker_default_1 (terminated by SIGKILL; not expected)
cvat | 2021-10-20 21:36:53,671 DEBG received SIGCHLD indicating a child quit
cvat | 2021-10-20 21:36:54,681 INFO spawned: 'rqworker_default_1' with pid 117
cvat | 2021-10-20 21:36:54,798 DEBG 'rqworker_default_1' stderr output:
cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout
cvat |
cvat | 2021-10-20 21:36:54,905 DEBG 'rqworker_default_1' stderr output:
cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds
cvat |
cvat | 2021-10-20 21:36:54,962 DEBG 'rqworker_default_1' stderr output:
cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device
cvat |
cvat | 2021-10-20 21:36:54,965 DEBG 'rqworker_default_1' stderr output:
cvat | bash: no job control in this shell
cvat |
cvat | 2021-10-20 21:36:55,966 INFO success: rqworker_default_1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
cvat | 2021-10-20 21:37:23,386 DEBG 'runserver' stderr output:
cvat | bash: [20: 1 (255)] tcsetattr: Inappropriate ioctl for device
cvat |
cvat | 2021-10-20 21:37:23,387 DEBG fd 40 closed, stopped monitoring <POutputDispatcher at 139783414089568 for <Subprocess at 139783414089520 with name runserver in state RUNNING> (stdout)>
cvat | 2021-10-20 21:37:23,387 DEBG fd 44 closed, stopped monitoring <POutputDispatcher at 139783413802128 for <Subprocess at 139783414089520 with name runserver in state RUNNING> (stderr)>
cvat | 2021-10-20 21:37:23,388 INFO exited: runserver (exit status 137; not expected)
cvat | 2021-10-20 21:37:23,388 DEBG received SIGCHLD indicating a child quit
cvat | 2021-10-20 21:37:23,401 INFO spawned: 'runserver' with pid 126
cvat | 2021-10-20 21:37:23,525 DEBG 'runserver' stderr output:
cvat | wait-for-it.sh: waiting for cvat_db:5432 without a timeout
cvat |
cvat_db | 2021-10-20 21:37:23.561 UTC [29] LOG: incomplete startup packet
cvat | 2021-10-20 21:37:23,586 DEBG 'runserver' stderr output:
cvat | wait-for-it.sh: cvat_db:5432 is available after 0 seconds
cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device
cvat | bash: no job control in this shell
cvat |
cvat | 2021-10-20 21:37:23,884 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:37:23,890 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 17
cvat |
cvat | 2021-10-20 21:37:23,892 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none)
cvat |
cvat | 2021-10-20 21:37:23,909 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:37:23,915 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 11
cvat |
cvat | 2021-10-20 21:37:24,422 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:37:24,423 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
cvat | 2021-10-20 21:37:24,442 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 17
cvat |
cvat | 2021-10-20 21:37:24,444 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none)
cvat |
cvat | 2021-10-20 21:37:24,461 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:37:24,473 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 11
cvat |
cvat | 2021-10-20 21:37:48,474 DEBG fd 39 closed, stopped monitoring <POutputDispatcher at 139783413801696 for <Subprocess at 139783414089328 with name rqworker_low in state RUNNING> (stderr)>
cvat | 2021-10-20 21:37:48,483 DEBG fd 35 closed, stopped monitoring <POutputDispatcher at 139783414089376 for <Subprocess at 139783414089328 with name rqworker_low in state RUNNING> (stdout)>
cvat | 2021-10-20 21:37:49,486 INFO exited: rqworker_low (terminated by SIGKILL; not expected)
cvat | 2021-10-20 21:37:49,487 DEBG received SIGCHLD indicating a child quit
cvat | 2021-10-20 21:37:50,492 INFO spawned: 'rqworker_low' with pid 152
cvat | 2021-10-20 21:37:50,627 DEBG 'rqworker_low' stderr output:
cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout
cvat |
cvat | 2021-10-20 21:37:50,736 DEBG 'rqworker_low' stderr output:
cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds
cvat |
cvat | 2021-10-20 21:37:50,759 DEBG 'rqworker_low' stderr output:
cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device
cvat |
cvat | 2021-10-20 21:37:50,760 DEBG 'rqworker_low' stderr output:
cvat | bash: no job control in this shell
cvat |
cvat | 2021-10-20 21:37:51,766 INFO success: rqworker_low entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
cvat | 2021-10-20 21:38:13,439 DEBG fd 29 closed, stopped monitoring <POutputDispatcher at 139783414259520 for <Subprocess at 139783414089040 with name rqworker_default_0 in state RUNNING> (stderr)>
cvat | 2021-10-20 21:38:13,440 DEBG fd 25 closed, stopped monitoring <POutputDispatcher at 139783414089088 for <Subprocess at 139783414089040 with name rqworker_default_0 in state RUNNING> (stdout)>
cvat | 2021-10-20 21:38:14,052 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:38:14,053 INFO exited: rqworker_default_0 (terminated by SIGKILL; not expected)
cvat | 2021-10-20 21:38:14,053 DEBG received SIGCHLD indicating a child quit
cvat | 2021-10-20 21:38:14,065 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 17
cvat |
cvat | 2021-10-20 21:38:14,068 INFO spawned: 'rqworker_default_0' with pid 163
cvat | 2021-10-20 21:38:14,069 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none)
cvat |
cvat | 2021-10-20 21:38:14,104 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:38:14,105 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 11
cvat |
cvat | 2021-10-20 21:38:14,200 DEBG 'rqworker_default_0' stderr output:
cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout
cvat |
cvat | 2021-10-20 21:38:14,255 DEBG 'rqworker_default_0' stderr output:
cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds
cvat |
cvat | 2021-10-20 21:38:14,257 DEBG 'rqworker_default_0' stderr output:
cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device
cvat |
cvat | 2021-10-20 21:38:14,257 DEBG 'rqworker_default_0' stderr output:
cvat | bash: no job control in this shell
cvat |
cvat | 2021-10-20 21:38:15,259 INFO success: rqworker_default_0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
cvat | 2021-10-20 21:38:15,580 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:38:15,595 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 17
cvat |
cvat | 2021-10-20 21:38:15,597 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none)
cvat |
cvat | 2021-10-20 21:38:15,624 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:38:15,641 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 11
cvat |
cvat | 2021-10-20 21:38:21,296 DEBG fd 19 closed, stopped monitoring <POutputDispatcher at 139783414258656 for <Subprocess at 139783414088656 with name git_status_updater in state RUNNING> (stderr)>
cvat | 2021-10-20 21:38:21,298 DEBG fd 15 closed, stopped monitoring <POutputDispatcher at 139783414088704 for <Subprocess at 139783414088656 with name git_status_updater in state RUNNING> (stdout)>
cvat | 2021-10-20 21:38:22,300 INFO exited: git_status_updater (terminated by SIGKILL; not expected)
cvat | 2021-10-20 21:38:22,301 DEBG received SIGCHLD indicating a child quit
cvat | 2021-10-20 21:38:23,305 INFO spawned: 'git_status_updater' with pid 181
cvat | 2021-10-20 21:38:23,423 DEBG 'git_status_updater' stderr output:
cvat | wait-for-it.sh: waiting for cvat_redis:6379 without a timeout
cvat |
cvat | 2021-10-20 21:38:23,482 DEBG 'git_status_updater' stderr output:
cvat | wait-for-it.sh: cvat_redis:6379 is available after 0 seconds
cvat | bash: cannot set terminal process group (-1): Inappropriate ioctl for device
cvat | bash: no job control in this shell
cvat |
cvat | 2021-10-20 21:38:24,486 INFO success: git_status_updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
cvat | 2021-10-20 21:38:32,396 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:38:32,397 INFO reaped unknown pid 149
cvat | 2021-10-20 21:38:32,397 DEBG received SIGCHLD indicating a child quit
cvat | 2021-10-20 21:38:32,415 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 17
cvat |
cvat | 2021-10-20 21:38:32,416 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_add_identity: add ssh-rsa SHA256:6HZvs3AmN4SolXRMJhO3mtSB7Rbm1QwAHIL8DjxKISo "django@3977e1f9cf92" (life: 0) (confirm: 0) (provider: none)
cvat |
cvat | 2021-10-20 21:38:32,441 DEBG 'ssh-agent' stderr output:
cvat | debug2: fd 4 setting O_NONBLOCK
cvat |
cvat | 2021-10-20 21:38:32,446 DEBG 'ssh-agent' stderr output:
cvat | debug1: process_message: socket 1 (fd=4) type 11
cvat |
cvat_ui | 172.28.0.2 - - [20/Oct/2021:21:38:43 +0000] "GET / HTTP/1.1" 200 1045 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:93.0) Gecko/20100101 Firefox/93.0" "172.28.0.1"
cvat_ui | 172.28.0.2 - - [20/Oct/2021:21:38:45 +0000] "GET /cvat-ui.f9340a4e583f3306fd17.min.js HTTP/1.1" 200 5955484 "http://localhost:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:93.0) Gecko/20100101 Firefox/93.0" "172.28.0.1"
cvat | 2021-10-20 21:38:45,441 INFO reaped unknown pid 151
cvat | 2021-10-20 21:38:45,448 DEBG received SIGCHLD indicating a child quit
cvat_ui | 172.28.0.2 - - [20/Oct/2021:21:38:45 +0000] "GET /favicon.ico HTTP/1.1" 200 102581 "http://localhost:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:93.0) Gecko/20100101 Firefox/93.0" "172.28.0.1"
cvat_ui | 172.28.0.2 - - [20/Oct/2021:21:38:51 +0000] "GET /download.worker.cb069004e9ace5f89e8e.js HTTP/1.1" 200 16721 "http://localhost:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:93.0) Gecko/20100101 Firefox/93.0" "172.28.0.1"
cvat | 2021-10-20 21:39:15,921 INFO reaped unknown pid 150
cvat | 2021-10-20 21:39:15,922 DEBG received SIGCHLD indicating a child quit

[bsekachev]

Hi Andrew,

What is docker ps output?

[azhavoro]

@AndrewOkie Based on cvat | 2021-10-20 21:37:23,388 INFO exited: runserver (exit status 137; not expected) I only guess there is out of memory issue, how much RAM does your VM have?

[AndrewOkie]

@AndrewOkie Based on cvat | 2021-10-20 21:37:23,388 INFO exited: runserver (exit status 137; not expected) I only guess there is out of memory issue, how much RAM does your VM have?

Yes, indeed, it is a small machine with 570 MiB of free RAM available. How much does CVAT require to work?

[azhavoro]

@AndrewOkie Based on cvat | 2021-10-20 21:37:23,388 INFO exited: runserver (exit status 137; not expected) I only guess there is out of memory issue, how much RAM does your VM have?

Yes, indeed, it is a small machine with 570 MiB of free RAM available. How much does CVAT require to work?

If you do not want to use any additional components such as analytics or run automatic annotations, I think 4-8 GB of total RAM will be enough.

[azhavoro]

I hope you solved the issue, don't hesitate to reopen if you still have questions.