Unusual gzip activity in docker version of DS after docker restart

[msaeedm]

Do you want to request a feature or report a bug?
Suspected bug

What is the current behavior?
DocumentServer logs are full of weird gzip activity. This is associated with medium to high CPU usage at rest. Here is a sample from the log file:

gzip: ./web-apps/apps/spreadsheeteditor/main/resources/help/en/Functions/multi.htm.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz compressed to ./web-apps/apps/spreadsheeteditor/main/resources/help/en/Functions/mul.ht.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz
gzip: ./web-apps/apps/spreadsheeteditor/main/resources/help/en/Functions/imsin.htm.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz compressed to ./web-apps/apps/spreadsheeteditor/main/resources/help/en/Functions/ims.ht.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz
gzip: ./web-apps/apps/spreadsheeteditor/main/resources/help/en/Functions/countbla.htm.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz compressed to ./web-apps/apps/spreadsheeteditor/main/resources/help/en/Functions/count.htm.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz.gz

Here is the full log for your review
https://www.dropbox.com/s/eofvmhzyhkmgoa2/ds.log?dl=0

There also seems to be intermittent restarts of services inside the container, an example of which can be seen toward the end of the log file. I am not sure if this is related to the gzip issue or not.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.
Simply running the container (does not matter if it is first time or not) causes this behavior

What is the expected behavior?
Absence of this apparent erratic gzip behavior within the container

Did this work in previous versions of DocumentServer?
I tried both 5.1.4.22 and 5.1.3.35 with same result

DocumentServer version:
Currently using 5.1.4.22

Operating System:
My docker host in on:
CentOS Linux 7 (Core) with Kernel: Linux 3.10.0-862.3.2.el7.x86_64

Browser version:
Firefox quantum 60.0.1

**More info
My Nginx vhost:

## Server for onlyoffice subdomain
server {
	listen 443 ssl http2;
	listen [::]:443 ssl;
	server_name onlyoffice.MYDOMAIN.me;
	server_tokens off;
	root /usr/share/nginx/html;
	gzip off;
	
	resolver 127.0.0.11 valid=30s ipv6=off;
	set $upstream https://onlyoffice:443;

	ssl_certificate /keys/server.crt;
	ssl_certificate_key /keys/server.key;

	add_header X-Content-Type-Options nosniff;


	location / {
		proxy_pass $upstream;
		proxy_redirect     off;

		proxy_set_header X-Forwarded-Proto $the_scheme;
		proxy_set_header X-Real-IP $remote_addr;

		proxy_set_header Host $http_host;

		proxy_set_header X-Forwarded-Host $the_host;

		proxy_read_timeout 1800s;
		proxy_connect_timeout 1800s;
	}
}

[ShockwaveNN]

@msaeedm Hi there, can you show me full command for running container and in which log file those errors are present.
Never seen anything like this before

[msaeedm]

@ShockwaveNN Hi, here is my docker command:

docker run -d \
--name=onlyoffice --restart=always \
--network=dockeronlyofficeowncloud_onlyoffice \
--memory=2g --memory-swap=2g \
--cpus=2 -c 512 \
--log-driver json-file --log-opt max-size=2m \
--label=com.centurylinklabs.watchtower.enable=false \
--expose 8000 \
--env-file /permfolders/onlyoffice.env \
-p 801:80 -p 4431:443 \
-v /permfolders/onlyoffice/logs:/var/log/onlyoffice \
-v /permfolders/onlyoffice/nginx/ds-ssl.conf:/etc/nginx/conf.d/ds-ssl.conf:ro \
-v /permfolders/onlyoffice/nginx/ds-nginx.conf:/etc/nginx/nginx.conf:ro \
-v /permfolders/onlyoffice/certs:/app/onlyoffice/DocumentServer/data/certs:ro \
-v /etc/localtime:/etc/localtime:ro \
onlyoffice-mod/documentserver:5.1.4.22

Here are the relevant files from the run command above in order to enable SSL and the secret key.
onlyoffice.env
ds-nginx.conf
ds-ssl.conf

Also, please note that I have changed "rejectUnauthorized": from true to false in /etc/onlyoffice/documentserver/default.json followed by supervisorctl restart all [this was needed as per this reported bug/issue ]. To avoid repeating this last step on each restart, I created an image onlyoffice-mod with the modified default.json file.

The logs mentioned in the original post were taken from docker logs onlyoffice

[ShockwaveNN]

@msaeedm could you attach to container and see if any file in /var/log/onlyoffice show those errors?

[msaeedm]

@ShockwaveNN I checked all log files under all /var/log/onlyoffice sub-directories. Interestingly, I do not see those errors anywhere there.

[ShockwaveNN]

@msaeedm Ok, I think those errors are from last line of this file https://github.com/ONLYOFFICE/Docker-DocumentServer/blob/master/run-document-server.sh#L339

Actually using docker logs is not a preferred way, we change this in next release, but until now logs should be watch by attaching to container, or mounting/var/log/onlyoffice as volume as you did.

[ShockwaveNN]

@msaeedm I tryed your docker run command but I cannot reproduce those logs entries

[msaeedm]

@ShockwaveNN If you want to reproduce it, you will need to give it time in order for those errors to appear in docker logs. I created another container yesterday and kept it running. I checked docker logs today and saw tons of those entries.

[ShockwaveNN]

@msaeedm maybe those errors from some real user or software that trying to access your server by unusual way (like incorrect adress or so).
I check some of my container that run 2 days ago and docker logs contain no such activtiy

[msaeedm]

@ShockwaveNN Hmmm..I suppose the next diagnostic step is for me to configure my environment and see what happens. I will create a VM, install docker and run this container and see what logs I get and report back here. Meanwhile, if you have any troubleshooting suggestions, I am all ears :)

[ShockwaveNN]

@msaeedm sorry have no idea about troubleshooting.
You can try simplify your docker run command, since it has A LOT of options and see if something changed, it can help pinpoint exact reason why this happened

[ShockwaveNN]

@msaeedm Thanks to @flaminestone we were able to reproduce this error by

docker run -itd --name onlyoffice onlyoffice/documentserver:5.1.4.22
sleep 60 # wait server to init
docker exec -it onlyoffice sh -c "ls /var/www/onlyoffice/documentserver/web-apps/apps/spreadsheeteditor/main/resources/help/en/Functions/fisher.htm.*"
# one entry fisher.htm.gz
docker restart onlyoffice
sleep 60 # wait server to init
docker exec -it onlyoffice sh -c "ls /var/www/onlyoffice/documentserver/web-apps/apps/spreadsheeteditor/main/resources/help/en/Functions/fisher.htm.*"
# two entry fisher.htm.gz and fisher.htm.gz.gz

To reproduce you may add one additional restart for solving problem with starting RabbitMQ (see ONLYOFFICE/Docker-DocumentServer#92)
Seems this new gz entries added after each restart and it's no good.
But after in your cases seems there were like 81 restart.

[msaeedm]

Thanks @ShockwaveNN @flaminestone

Not sure if this helps but I tested a separate RabbitMQ container and configured onlyoffice DS container to connect to it. Restarting the DS container STILL shows the gz issue even though RabbitMQ was not restarted (it is running in a different container)

[ShockwaveNN]

@msaeedm I meant, that your error have nothing to do with RabbitMQ bug, I added this note so our dev team have no question about it.

[msaeedm]

Ah ok. I misunderstood.

[agolybev]

Hi @msaeedm, the problem with gziping will be fixed in next hotfix (v5.1.5)

[msaeedm]

Great @agolybev . Thank you and the dev team for your efforts

[ShockwaveNN]

Checked onlyoffice/4testing-documentserver:5.1.5.35 and there is no problems with gzip

[ShockwaveNN]

5.1.5 released yesterday, this issue should be fixed