AVX is required to run CVAT (should be optional)

[ghost]

My actions before raising this issue

• Read/searched the docs
• Searched past issues

Right after logging in a error message pops up saying: Could not get formats from the server. The network console shows that the api request https://mydomain.net/api/v1/server/dataset/formats fails with status code 500.

Dumping annotations only works in CVAT XML 1.1, LabelMe Zip 3.0 and MOT CSV 1.0 format. I guess this is due to the error mentioned above. For instance, trying to dump annotations in COCO Json 1.0 format showed the message: Can not dump annotations for the task. Code: 500. Message: "Work-horse process was terminated unexpectedly (waitpid returned 132)"

Expected Behaviour

No error message being displayed. Dumping annotations works.

Current Behaviour

The opposite of the expected behaviour.

Possible Solution

I thought the lines:

2020-04-10 17:52:43,603 DEBG 'runserver' stderr output:
[Fri Apr 10 17:52:43.602004 2020] [wsgi:error] [pid 159:tid 140124450965248] [client 172.31.0.6:47762] Truncated or oversized response headers received from daemon process 'localhost:8080': /tmp/mod_wsgi-localhost:8080:1000/htdocs/api, referer: https://mydomain.net/auth/login

in the logs were connected to the problem and found something about wsgi and numpy-like libraries not being able to handle requests from multiple threads. Since I am unfamiliar with wsgi, django and the whole project I just tried my best and added WSGIApplicationGroup %{GLOBAL} in the mod_wsgi.conf file and rebuilt and restarted the images which didn't help of course.

Steps to Reproduce and Context

Unfortunately I don't know how you could reproduce. I just installed everything as suggested in the installation guide and started up the containers. I did the same on my windows machine where everything works fine so far. The only difference is that my installation on my Ubuntu server is being served from a nginx reverse proxy with a simple single proxy_pass statement.

Your Environment

• commit 39eb9c5 (HEAD -> master, origin/master, origin/HEAD)
• Docker 19.03.8
• Ubuntu Server 18.04 LTS

Logs

cvat-log-after-error-message.txt
cvat_db-log-after-error-message.txt

[azhavoro]

Hi, could you please check that your cpu supports AVX instructions: search for "avx" in lscpu or cat /proc/cpuinfo output

[ghost]

Hi, could you please check that your cpu supports AVX instructions: search for "avx" in lscpu or cat /proc/cpuinfo output

Well, it does not. If that is actually the problem than would it be nice if there was a requirements section in the installation guide of some kind.

Tensorflow is the only package I know requiring AVX since 1.6, so I changed tensorflow<1.6.0 in cvat's base requirements file and rebuilt the images. Unfortunately that did not fix the problem but it was just a best guess anyways.

[zhiltsov-max]

The only occurrence of the error with failing formats requests we met was related to AVX usage in tensorflow initialization. Try to check if complete removal of this package helps.

We've also experienced similar problem with dumping when the server has been running out of memory. Can it be the problem in your case?

[ghost]

Thank you very much. I removed tensorflow and keras and it seems to work so far. No error messages and dumping annotations works (except for the for the TFRecord format which makes sense, I guess).

I removed keras as well because I thought it might install tensorflow automatically and since I didn't see any other backends for it I thought it's not needed anyways. But I don't actually know if I had to remove keras, too.

I'm going to close this issue since my problem and the cause of it have been solved. I'd still suggest adding a hint somewhere that AVX must be supported by the CPU.

[nmanovic]

It seems that a lot of users have the problem. Need to workaround it somehow or have a clear message to avoid the confusion.

[nmanovic]

@zhiltsov-max , is it visible to disable tfrecord format if AVX isn't supported?

[NormanTUD]

From what I see, the main dependency that requires AVX is TF==2.8.0.

There are non-AVX-compiles availabe, e.g. may

RUN python3 -m pip install --upgrade pip
RUN python3 -m pip install wheel
RUN python3 -m pip install https://tf.novaal.de/btver1/tensorflow-2.8.0-cp38-cp38-linux_x86_64.whl

in the Dockerfile to install it and re-build the container (docker build --no-cache .) helps.

[NormanTUD]

For me, it didn't ever work and I must sadly say that I abandoned CVAT because of that. It looked very nice and I'm sure works great when it works, but installing it never did for me.
For anyone like me who's just interested in a free and quick solution I created a very simple PHP page that allows many people to annotate datasets via mouse with groups of people and then get the results. It's alpha still!, but for me it works better than something great that I cannot access for some technical reason.

https://github.com/NormanTUD/SimpleImageAnnotationTool

Maybe this helps someone someday.